Introduction

This is small collection of code and various other snippets that I've find useful to refer back to over the years. I try to keep it up-to-date with anything that I do multiple times.

Ansible

Run role against arbitrary host

From here. Note the comma after the IP or FQDN as described here. Additional hosts can be added by supplying -i parameter with more arguments (comma at the end only if total count is 1).

ansible-playbook -i '<IP or FQDN>,' -u '<user name>' --extra-vars \
  'ansible_winrm_server_cert_validation=ignore \
  ansible_connection=winrm \
  ansible_winrm_transport=credssp \
  ansible_password=<password>' --tags '<tag value for a role>' playbook.yml

Run ad-hoc command against arbitrary host

Replace the final 'all' with a more precise host pattern if you passed more than one IP or FQDN to the initial list (comma at the end only if total count is 1).

ansible -i '<IP or FQDN>,' -u '<user name>' --extra-vars \
  'ansible_winrm_server_cert_validation=ignore \
  ansible_connection=winrm \
  ansible_winrm_transport=credssp \
  ansible_password=<password>' -m 'win_shell' -a 'ipconfig' 'all'

Add timing information to playbook execution output

The below is from ansible.cfg. From here.

[defaults]
callback_whitelist = profile_tasks

Make verbose output more readable by using YAML instead of JSON

ANSIBLE_STDOUT_CALLBACK='yaml'

Debug variables without running entire playbook

ansible -m debug <host> -a "var=hostvars[inventory_hostname].<variable>"

Docker/Podman

Parse lines from an arguments file to separate parameters for building

From here

podman build -t foo \
  $(while IFS= read -r line; do args+="--build-arg ${line} "; \above
  done < <(cat .arg); \
echo "$args"; unset args) .

Remove all 'exited' containers

podman rm $(podman ps --all -q -f status=exited)

Build and run container based on Dockerfile in current context

podman build -t foo . && podman run --rm -it foo

Prune everything that shouldn't exist anymore without any confirmation

podman system prune -a -f

Remove all images except `latest`

podman images | grep -v "latest" | tail -n +2 | awk '{ print $3 }' | xargs --no-run-if-empty podman rmi

Possible improvement when executing `RUN` within a Dockerfile

Benefit is that when a specific line fails, then the error message is much more concise as opposed to the standard method of using ampersands.

RUN set -eu; \
    python3 -m venv venv; \
    venv/bin/pip install -r requirements.txt; \
    venv/bin/pip install -r requirements-dev.txt; \
    echo 'Venv creation + requirements installation: OK';

Remove dangling `<none>:<none>` images

docker rmi $(docker images -f "dangling=true" -q)

Git

Push existing repository to new remote

git remote add <name of new remote> <HTTPS or SSH URL>
git push <name of new remote> master

Pretty-print branch graph

git log --all --decorate --oneline --graph

Move `master` back X amount of commits

git checkout master
git reset --hard <old_commit_id>
git push -f origin master

Replay changes on `master` to some other branch

Beware of blindly accepting any incoming changes in favor of your own. From here.

git checkout master
git pull
git checkout different_branch
git rebase -Xtheirs master
git push --force

Show changed files in specified commit hash

From here.

git diff-tree --no-commit-id --name-only <commit hash>

Create patch file from diff

git diff file.json > file.patch

Create patch file from commit

git show <commit hash> > commit.patch

Apply patch file

git apply commit.patch

Bulk create patch files from individual files when running `git diff` in a repository

From here.

OLDIFS=$IFS; IFS=';' \
  blocks=$(git diff | sed -n '/diff/,/(diff|$)/ {/diff / s/^/\;/; p}'); \
  for block in ${blocks#;}; do \
    echo "$block" > $(echo "$block" | head -n 1 | rev | cut -d "/" -f 1 | rev).patch; \
  done; \
IFS=$OLDIFS

Show diff of stashed hunk

git stash show -p [stash@{N}]

Bulk create separate stashes of every changed file with a message equaling the filename

git status -s | cut -d " " -f 3 | xargs -I {} git stash push {} -m "{}"

Pop every entry from the stash back to the working tree

git stash list | cut -d ":" -f 1 | xargs -I {} git stash pop

Move unpushed commits to a new branch

Pull latest changes from 'origin/master' if haven't already. From here.

git checkout -b new_branch
git checkout master
git reset --hard origin/master

Copy commit to current branch

git cherry-pick <commit hash>

Undo pushed commit that nobody has yet pulled

git reset HEAD^ --hard
git push --force origin

View history of specific function in file

git log -L :<function>:<file>

Speed up Git for larger repositories

git config feature.manyFiles 1

Search through history for a specific word

git rev-list --all | ( while read revision; do git grep -F 'word' "$revision"; done; )

Delete remote branch

git push origin --delete branch/name

Bulk reset author of multiple (unpushed) commits (e.g. 9)

Set correct user name and email prior to this.

git rebase --onto HEAD~9 --exec "git commit --amend --reset-author --no-edit" HEAD~9

Re-order commits

Oldest commit will be at the top. Move commit down with ddp. Move commit up with ddkP

git rebase --interactive

Search for 'something' in a commit message

git log --all -i --grep='something'

Search for 'something' through all commits' contents

git grep 'something' $(git rev-list --all)

Clean new untracked files and directories

-d: recurse into directories as well
-f: go ahead with deletion
-n: dry-run

$ git clean -dn
Would remove mdbook
Would remove public/snippets/

$ git clean -df
Removing mdbook
Removing public/snippets/

Kubernetes

These are currently specific to kubectl and there are tons more here.

Set up short names for contexts

Open ~/.kube.config
Modify contexts section:

contexts:
- context:
    cluster: cluster1
    user: cluster1
  name: dev
- context:
    cluster: cluster2
    namespace: namespace1
    user: cluster2
  name: namespace1
- context:
    cluster: cluster3
    namespace: db
    user: cluster3
  name: prod
current-context: prod

Just use K9s

Here.

Use specific context (i.e. cluster)

kubectl config use-context <context>

List namespaces

kubectl get ns

Set namespace

This will also update the relevant context's namespace value in the ~/.kube/config file.

kubectl config set-context --current --namespace=<namespace>

List everything

This doesn't actually list everything. See below for more.

kubectl get all

Actually list everything. May take awhile.

function k_get_all () {
    for i in $(kubectl api-resources --verbs=list --namespaced -o name | grep -v "events.events.k8s.io" | grep -v "events" | sort | uniq); do
        kubectl get --show-kind --ignore-not-found ${i};
    done
}

Show pod logs

kubectl logs pod/<pod>

Miscellaneous

Create colored prompt with Git branch for Windows terminals

For example ConEmu.

PS1='${debian_chroot:+($debian_chroot)}\[\033[01;35m\] \w\[\033[01;33m\]$(__git_ps1)\[\033[01;m\] > '

JMESPath query differences

Query in JMESPath website:

[?tags.currently-used == 'False' || tags.currently_used == 'False'].name

Same query in jpterm:

[?tags."currently-used" == 'False' || tags."currently_used" == 'False']

Same query in console:

"[?tags.\"currently-used\" == 'False' || tags.\"currently_used\" == 'False'].name"

Convert x265 MP4 to x264

ffmpeg -i input.mp4 -c:v libx264 -crf 20 -c:a copy output.mp4`

Create Windows install media in Linux

Download WoeUSB
Download Windows 10 ISO
Find out USB's disk: sudo fdisk --list
Wipe disk entirely and burn ISO: sudo ./woeusb-5.2.4.bash --target-filesystem NTFS --device Win10_22H2_EnglishInternational_x64.iso /dev/sda

Install specific version of Flatpak package

Find relevant commit:

$ flatpak remote-info --log flathub org.signal.Signal
        ID: org.signal.Signal
       Ref: app/org.signal.Signal/x86_64/stable
      Arch: x86_64
    Branch: stable
Collection: org.flathub.Stable
  Download: 150,0 MB
 Installed: 422,3 MB
   Runtime: org.freedesktop.Platform/x86_64/22.08
       Sdk: org.freedesktop.Sdk/x86_64/22.08

    Commit: 7a69cbffa53f8d4962a0a15cfd4c941537c9d2aab7e5175757848f3985adf35c
    Parent: 0a4d650514861c74f5a3a559c9116104922b9aeb4cdcaedc1b364d97b158031a
   Subject: Update signal-desktop.deb to 6.16.0 (8c5276b3)
      Date: 2023-04-26 03:05:43 +0000
   History: 

    Commit: 0a4d650514861c74f5a3a559c9116104922b9aeb4cdcaedc1b364d97b158031a
   Subject: Remove unnecessary permissions (#438) (3f78ec79)
      Date: 2023-04-25 08:09:46 +0000

    Commit: 92c53789a85b84f79248e03ddc5fb6c664f17ef988e98fc196d157d5d77a73bf
   Subject: Update org.signal.Signal.metainfo.xml (74c6adfa)
      Date: 2023-04-20 10:47:54 +0000

    ...

Install it:

$ flatpak update --commit=0a4d650514861c74f5a3a559c9116104922b9aeb4cdcaedc1b364d97b158031a org.signal.Signal
Looking for updates…

        ID                         Branch         Op         Remote          Download
 1. [✓] org.signal.Signal          stable         u          flathub         62,2 MB / 166,6 MB

Updates complete.

Download audio and video using 'youtube-dl'

List available formats for a video:

$ youtube-dl -F https://www.youtube.com/watch?v=H5ejv-dTAaU
[youtube] H5ejv-dTAaU: Downloading webpage
[info] Available formats for H5ejv-dTAaU:
format code  extension  resolution note
249          webm       audio only tiny   50k , webm_dash container, opus  (48000Hz), 8.42MiB
250          webm       audio only tiny   60k , webm_dash container, opus  (48000Hz), 10.01MiB
251          webm       audio only tiny  109k , webm_dash container, opus  (48000Hz), 18.14MiB
140          m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2 (44100Hz), 21.54MiB
160          mp4        256x144    144p   22k , mp4_dash container, avc1.4d400c, 24fps, video only, 3.76MiB
278          webm       256x144    144p   57k , webm_dash container, vp9, 24fps, video only, 9.51MiB
133          mp4        426x240    240p   42k , mp4_dash container, avc1.4d4015, 24fps, video only, 7.12MiB
242          webm       426x240    240p   62k , webm_dash container, vp9, 24fps, video only, 10.39MiB
134          mp4        640x360    360p   76k , mp4_dash container, avc1.4d401e, 24fps, video only, 12.77MiB
243          webm       640x360    360p  124k , webm_dash container, vp9, 24fps, video only, 20.72MiB
135          mp4        854x480    480p  111k , mp4_dash container, avc1.4d401e, 24fps, video only, 18.47MiB
244          webm       854x480    480p  189k , webm_dash container, vp9, 24fps, video only, 31.49MiB
136          mp4        1280x720   720p  165k , mp4_dash container, avc1.4d401f, 24fps, video only, 27.51MiB
247          webm       1280x720   720p  331k , webm_dash container, vp9, 24fps, video only, 55.13MiB
137          mp4        1920x1080  1080p  627k , mp4_dash container, avc1.640028, 24fps, video only, 104.35MiB
248          webm       1920x1080  1080p  633k , webm_dash container, vp9, 24fps, video only, 105.45MiB
18           mp4        640x360    360p  343k , avc1.42001E, 24fps, mp4a.40.2 (44100Hz), 57.20MiB
22           mp4        1280x720   720p  294k , avc1.64001F, 24fps, mp4a.40.2 (44100Hz) (best)

Combine audio-only and video-only formats, and start downloading

$ youtube-dl -f 248+251 https://www.youtube.com/watch?v=H5ejv-dTAaU
[youtube] H5ejv-dTAaU: Downloading webpage
[dashsegments] Total fragments: 11
[download] Destination: The Death of Globalization!-H5ejv-dTAaU.f248.webm
[download] 100% of 105.45MiB in 01:32
[dashsegments] Total fragments: 2
[download] Destination: The Death of Globalization!-H5ejv-dTAaU.f251.webm
[download] 100% of 18.14MiB in 00:47
[ffmpeg] Merging formats into "The Death of Globalization!-H5ejv-dTAaU.webm"
Deleting original file The Death of Globalization!-H5ejv-dTAaU.f248.webm (pass -k to keep)
Deleting original file The Death of Globalization!-H5ejv-dTAaU.f251.webm (pass -k to keep)

PowerShell

Get today's date and time where time is set to midnight

[DateTime]::Today

Show list of logged on users

query user /server:$SERVER

Log off user by specifying session ID

logoff <session ID>

Reload a local module

Import-Module -Name .\module.psd1 -Force

Pretty-print minified JSON

$String | ConvertFrom-Json | ConvertTo-Json -Depth 100

Convert from Base64

[Text.Encoding]::Utf8.GetString([Convert]::FromBase64String($String))

Convert string to boolean

[System.Convert]::ToBoolean($String)

Resolve FQDN

[System.Net.Dns]::GetHostByName($FQDN)

Shell

Basic `for` loop to iterate over lines in a file

for pkg in $(cat pkgs.txt); do sudo apt purge "$pkg" -y; done

More complex `for` loop using `if` statement

Useful for control actions, cleaning up output, etc.

for node in $(cat nodes.txt); do \
  echo "Node: ${node}"; \
  ssh -q -t "$node" 'if [[ $(lsblk | grep -i lvm) ]]; then sudo apt install mdadm -y; fi'; \
done

Checking very busy log files for their contents

This does not hang your console as opposed to using tail -f.

watch -n 0.5 sudo tail /var/log/named/queries.log

Alternative conditional logic in `for` loop iterating over array variable

Declare nodes variable separately or prepend to loop and separate with semicolon.

for node in "${nodes[@]}"; do \
  ping -c 2 -W 0.1 "$node" > /dev/null && \
  echo "OK: ${node}" || echo "NOT OK: ${node}"; \
done

Use `while` loop to iterate over lines in a file

Avoids calls to cat as is the case with the for loop example. Using madison command rather than policy seems to be slightly faster.

while read pkg; do \
  if [[ $(apt-cache madison "$pkg") ]]; then \
    echo "OK: ${pkg} exists in some repo"; \
  else \
    echo "NOT OK: ${pkg} doesn't exist in any repo"; \
  fi; \
done < pkgs.txt

Match lines into an array

types=($(grep -oE 'pattern' input.txt))

Grab block of text between two patterns

sed -n '/pattern1/,/pattern2/p' input.txt

Just see octal permissions for a file or directory

stat -c '%a' /etc/passwd

Grab last character from string

last_character=${string_variable:-1}

Parse file list from output of `grep` into subsequent commands

grep -rl '\-\- MARK \-\-' /var/log/* | \
  while read line; do \
    echo "Working with file '${line}'"; \
    grep MARK "$line" | tail -n1; \
  done

Include lines before and after a `grep` match

grep -B 3 -A 3 -i "hv_fcopy" /var/log/messages

Find all unique directories in listed directories that contain files modified 10 minutes ago since the command was ran

ls | xargs -I {} find {} -type f -mmin -10 | cut -d "/" -f2 | sort -u

Find all files in the current directories that were modified at least a minute ago, are larger than 500MB, and long list them

find . -type f -mmin -1 -size +500M -exec ls -lsh {} \;

Find all files in the current directories that were modified at least a day ago, are larger than 2GB, and empty their contents

find . -type f -mtime -1 -size +2G -exec bash -c 'echo > {}' \;

Run arbitrary command against a list of directories

ls | xargs -I {} git -C {} pull

Step-by-step debug Bash scripts

Move ahead with Enter key.

set -x
trap read debug

Change timezone interactively

dpkg-reconfigure tzdata

Search binary file that looks like text while ignoring case

grep -ai "end:" /var/log/syslog

Count time, calls, and errors for each system call when performing a directory listing

strace -c ls test/

Add to script to determine which line number the execution is at

echo "DEBUG: ${LINENO}"

Remove duplicated lines from a file without messing up the order

awk '!visited[$0]++' your_file > deduplicated_file

Run local script on a remote endpoint using SSH

ssh -q <username>@<endpoint> "sudo bash -s" < local_script.sh

Create new directory and change right into it

Oneliner

mkdir new_directory && cd $_

Alias

From here.

mkcd () {
  mkdir "$1"
  cd "$1"
}

Recall argument to last used command

From here.

$_
!$
Alt + .
!:1
!:1-2

Get SSH key fingerprint

SHA-256

ssh-keygen -lf ~/.ssh/id_rsa.pub

MD5

ssh-keygen -E md5 -lf ~/.ssh/id_rsa.pub

Find broken symbolic links in current directory

find . -xtype l

Bulk fix relative symbolic links

find . -lname '<relative-to-source target>*' \
  -exec sh -c 'ln -sfn "<new relative-to-source target>/$(basename $0)" $0' {} \;

Run remote script on remote endpoint using SSH

ssh -q <username>@<endpoint> './location/to/script'

Create ISO from directory without truncating long names (`-l`)

Also by not replacing hyphens with underscores (-iso-level 4).

genisoimage -o data.iso -iso-level 4 -R -l data/

List ISO file contents without having to mount it

isoinfo -l -i data.iso

Simple colouring for log files, both static and running output

From here.

cat test.log | perl -pe 's/^\[\*\].*/\e[0;36m$&\e[0m/g; s/^\[\+\].*/\e[0;32m$&\e[0m/g; s/^\[\!\].*/\e[0;31m$&\e[0m/g'

Suppress Python warnings

For situations like these.

export PYTHONWARNINGS='ignore'

Remove last column in string based on delimiter

$ string='my_underscored_string_12345'
$ echo "$string" | rev | cut -d '_' -f 2- | rev
my_underscored_string

Prefix aliased command with backslash to avoid triggering alias

$ halt -p
REALLY!? -p
$ alias halt
alias halt='echo "REALLY!?"'
$ \halt -p
Connection to example.com closed by remote host.

Pretty print CSV files

From here

function pretty_csv {
    perl -pe 's/((?<=,)|(?<=^)),/ ,/g;' "$@" | column -t -s, | less  -F -S -X -K
}

$ pretty_csv data.csv
$ pretty_csv < data.csv
$ sort data.csv | pretty_csv

Pretty print TSV files

function pretty_tsv {
    perl -pe 's/((?<=\t)|(?<=^))\t/ \t/g;' "$@" | column -t -s $'\t' | less  -F -S -X -K
}

$ pretty_tsv data.tsv
$ pretty_tsv < data.tsv
$ sort data.tsv | pretty_tsv

Diff two files and save unified output to file

diff -u file1 file2 > files.diff

Show build information for cloud-based image

$ cat /etc/cloud/build.info 
build_name: server
serial: 20201211.1

Show top disk usage and exclude certain directories under root

du -Sh / --exclude=/{proc,sys,dev,var} | sort -rh | head -n 10

Use the built-in `:` as a short-hand for an infinite loop

while :; do "looping"; done

Re-execute a Bash to 'unsource' variables and aliases

exec /bin/bash

Use binary version of `time` instead of Bash built-in

This provides access to more information. From here.

$(which time) --verbose echo "test"

Use `perf stat` to easily repeat a command

Also provides additional useful measurements. More examples here

perf stat --null --repeat 5 --table echo "test"

Change nested key value in an array of JSON objects with `jq`

.parameters.vmObjects.value |= map(if .vmName == "router" then .moduleSnapshot = "fixed" else . end)

Use indirect references to use dynamic variable names

for host in "${hosts[@]}"; do
  declare "parent_disk_${host}=$parent_disk"
done

for host in "${hosts[@]}"; do
  parent_disk="parent_disk_${host}"
  echo "${!parent_disk}"
done

Test terminal's colors

msgcat --color=test

Bulk rename files in place

find . -type f -name '<file name>' -execdir mv {} "description.txt" \;

Encode with Base64 on a single line

echo "text" | base64 -w 0

Convert PEM to single-line

awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' combined.pem

Install requirements for Poetry using existing `requirements.txt`

cat requirements.txt | xargs poetry add

Direct standard output to a file in a directory that might not yet exist

echo "something" | install -D /dev/stdin directory/file.txt

Find and delete files older than 1 year

find /the/dir/to/start/in -type f -mtime +365 -ls -exec rm -f -- {} \;

View permissions as a tree

-p: permissions
-u: username/userid
-f: full path
-i: don't print indentation lines
-d: print directories only

tree -pufid

Bulk uninstall `pip` packages according to a wildcard

pip freeze | grep "azure*" | xargs -n 1 pip uninstall -y

Show transaction history for a package

dnf history list <package>

Show information about specific transaction in history

Undo last transaction

dnf history undo last

List SystemD timers

systemctl list-timers

Show execution of service tied to a timer of an identical name

journalctl -u name.timer
journalctl -u name.service

Copy remote directory to local system

scp -rCp <user>@<remote>:<remote path> <local path>
# OR (faster)
rsync -azvhP <user>@<remote>:<remote path> <local path>

Overwrite existing directory with contents from another

rsync -av --delete ~/new/ ~/old

Count number of installed kernels

$ sudo dnf list --installed kernel-core* | tail -n +2 | wc -l
9

Increase number of installed kernels in `/etc/dnf/dnf.conf`

...
installonly_limit=10
...

Pin specific kernel version

$ sudo dnf install python3-dnf-plugins-extras-versionlock
$ # List kernel packages
$ rpm -qa kernel
kernel-6.0.18-300.fc37.x86_64
kernel-6.1.7-200.fc37.x86_64
kernel-6.1.8-200.fc37.x86_64
$ Add pin
$ sudo dnf versionlock add kernel-6.0.18-300.fc37.x86_64
Last metadata expiration check: 3:51:11 ago on E 30 jaan  2023 15:47:21.
Adding versionlock on: kernel-0:6.0.18-300.fc37.*
$ # Remove pin
$ sudo dnf versionlock delete kernel-6.0.18-300.fc37.x86_64
...

Undo ad hoc changes made to a SystemD service

For example systemd-resolved.

$ systemctl revert systemd-resolved.service
Removed "/etc/systemd/system/systemd-resolved.service.d/override.conf".
Removed "/etc/systemd/system/systemd-resolved.service.d".
$ systemctl restart systemd-resolved.service

Back up a file using brace expansion

Equivalent to cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak. More on brace expansion.

cp /etc/ssh/sshd_config{,.bak}

The same can be applied for directories:

cp -aR public{,.bak}

Restore a backed up file

Equivalent to cp /etc/ssh/sshd_config.bak /etc/ssh/sshd_config.

cp /etc/ssh/sshd_config{.bak,}

Download older version of a kernel

More information about Koji
A list of all kernels built for Fedora
A list of downloadable kernels built for Fedora
- Some may be removed

Fedora Discussion here.

koji download-build --arch=x86_64 <kernel package name>

Vim

Write file opened as 'readonly'

:w !sudo tee "%"

Visual editing can be used to delete any number of lines

Shift+V from point of cursor and press D key to delete

Search and replace

:%s/search/replace/g

Disable search highlight

:noh

Snippets