Skip to content
Back to home page

Shell

Basic for loop to iterate over lines in a file

for pkg in $(cat pkgs.txt); do sudo apt purge "$pkg" -y; done

More complex for loop using if statement

Useful for control actions, cleaning up output, etc.

for node in $(cat nodes.txt); do \
  echo "Node: ${node}"; \
  ssh -q -t "$node" 'if [[ $(lsblk | grep -i lvm) ]]; then sudo apt install mdadm -y; fi'; \
done

Checking very busy log files for their contents

This does not hang your console as opposed to using tail -f.

watch -n 0.5 sudo tail /var/log/named/queries.log

Alternative conditional logic in for loop iterating over array variable

Declare nodes variable separately or prepend to loop and separate with semicolon.

for node in "${nodes[@]}"; do \
  ping -c 2 -W 0.1 "$node" > /dev/null && \
  echo "OK: ${node}" || echo "NOT OK: ${node}"; \
done

Use while loop to iterate over lines in a file

Avoids calls to cat as is the case with the for loop example. Using madison command rather than policy seems to be slightly faster.

while read pkg; do \
  if [[ $(apt-cache madison "$pkg") ]]; then \
    echo "OK: ${pkg} exists in some repo"; \
  else \
    echo "NOT OK: ${pkg} doesn't exist in any repo"; \
  fi; \
done < pkgs.txt

Match lines into an array

types=($(grep -oE 'pattern' input.txt))

Grab block of text between two patterns

sed -n '/pattern1/,/pattern2/p' input.txt

Just see octal permissions for a file or directory

stat -c '%a' /etc/passwd

Grab last character from string

last_character=${string_variable:-1}

Parse file list from output of grep into subsequent commands

grep -rl '\-\- MARK \-\-' /var/log/* | \
  while read line; do \
    echo "Working with file '${line}'"; \
    grep MARK "$line" | tail -n1; \
  done

Include lines before and after a grep match

grep -B 3 -A 3 -i "hv_fcopy" /var/log/messages

Find all unique directories in listed directories that contain files modified 10 minutes ago since the command was ran

ls | xargs -I {} find {} -type f -mmin -10 | cut -d "/" -f2 | sort -u

Find all files in the current directories that were modified at least a minute ago, are larger than 500MB, and long list them

find . -type f -mmin -1 -size +500M -exec ls -lsh {} \;

Find all files in the current directories that were modified at least a day ago, are larger than 2GB, and empty their contents

find . -type f -mtime -1 -size +2G -exec bash -c 'echo > {}' \;

Run arbitrary command against a list of directories

ls | xargs -I {} git -C {} pull

Step-by-step debug Bash scripts

Move ahead with Enter key.

set -x
trap read debug

Change timezone interactively

dpkg-reconfigure tzdata

Search binary file that looks like text while ignoring case

grep -ai "end:" /var/log/syslog

Count time, calls, and errors for each system call when performing a directory listing

strace -c ls test/

Add to script to determine which line number the execution is at

echo "DEBUG: ${LINENO}"

Remove duplicated lines from a file without messing up the order

awk '!visited[$0]++' your_file > deduplicated_file

Run local script on a remote endpoint using SSH

ssh -q <username>@<endpoint> "sudo bash -s" < local_script.sh

Create new directory and change right into it

Oneliner

mkdir new_directory && cd $_

Alias

From here.

mkcd () {
  mkdir "$1"
  cd "$1"
}

Recall argument to last used command

From here.

$_
!$
Alt + .
!:1
!:1-2

Get SSH key fingerprint

SHA-256

ssh-keygen -lf ~/.ssh/id_rsa.pub

MD5

ssh-keygen -E md5 -lf ~/.ssh/id_rsa.pub
find . -xtype l
find . -lname '<relative-to-source target>*' \
  -exec sh -c 'ln -sfn "<new relative-to-source target>/$(basename $0)" $0' {} \;

Run remote script on remote endpoint using SSH

ssh -q <username>@<endpoint> './location/to/script'

Create ISO from directory without truncating long names (-l)

Also by not replacing hyphens with underscores (-iso-level 4).

genisoimage -o data.iso -iso-level 4 -R -l data/

List ISO file contents without having to mount it

isoinfo -l -i data.iso

Simple colouring for log files, both static and running output

From here.

cat test.log | perl -pe 's/^\[\*\].*/\e[0;36m$&\e[0m/g; s/^\[\+\].*/\e[0;32m$&\e[0m/g; s/^\[\!\].*/\e[0;31m$&\e[0m/g'

Suppress Python warnings

For situations like these.

export PYTHONWARNINGS='ignore'

Remove last column in string based on delimiter

$ string='my_underscored_string_12345'
$ echo "$string" | rev | cut -d '_' -f 2- | rev
my_underscored_string

Prefix aliased command with backslash to avoid triggering alias

$ halt -p
REALLY!? -p
$ alias halt
alias halt='echo "REALLY!?"'
$ \halt -p
Connection to example.com closed by remote host.

Pretty print CSV files

From here

function pretty_csv {
    perl -pe 's/((?<=,)|(?<=^)),/ ,/g;' "$@" | column -t -s, | less  -F -S -X -K
}

$ pretty_csv data.csv
$ pretty_csv < data.csv
$ sort data.csv | pretty_csv

Pretty print TSV files

function pretty_tsv {
    perl -pe 's/((?<=\t)|(?<=^))\t/ \t/g;' "$@" | column -t -s $'\t' | less  -F -S -X -K
}

$ pretty_tsv data.tsv
$ pretty_tsv < data.tsv
$ sort data.tsv | pretty_tsv

Diff two files and save unified output to file

diff -u file1 file2 > files.diff

Show build information for cloud-based image

$ cat /etc/cloud/build.info 
build_name: server
serial: 20201211.1

Show top disk usage and exclude certain directories under root

du -Sh / --exclude=/{proc,sys,dev,var} | sort -rh | head -n 10

Use the built-in : as a short-hand for an infinite loop

while :; do "looping"; done

Re-execute a Bash to ‘unsource’ variables and aliases

exec /bin/bash

Use binary version of time instead of Bash built-in

This provides access to more information. From here.

$(which time) --verbose echo "test"

Use perf stat to easily repeat a command

Also provides additional useful measurements. More examples here

perf stat --null --repeat 5 --table echo "test"

Change nested key value in an array of JSON objects with jq

.parameters.vmObjects.value |= map(if .vmName == "router" then .moduleSnapshot = "fixed" else . end)

Use indirect references to use dynamic variable names

for host in "${hosts[@]}"; do
  declare "parent_disk_${host}=$parent_disk"
done

for host in "${hosts[@]}"; do
  parent_disk="parent_disk_${host}"
  echo "${!parent_disk}"
done

Test terminal’s colors

msgcat --color=test

Bulk rename files in place

find . -type f -name '<file name>' -execdir mv {} "description.txt" \;

Encode with Base64 on a single line

echo "text" | base64 -w 0

Convert PEM to single-line

awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' combined.pem

Install requirements for Poetry using existing requirements.txt

cat requirements.txt | xargs poetry add

Direct standard output to a file in a directory that might not yet exist

echo "something" | install -D /dev/stdin directory/file.txt

Find and delete files older than 1 year

find /the/dir/to/start/in -type f -mtime +365 -ls -exec rm -f -- {} \;

View permissions as a tree

  • -p: permissions
  • -u: username/userid
  • -f: full path
  • -i: don’t print indentation lines
  • -d: print directories only
tree -pufid

Bulk uninstall pip packages according to a wildcard

pip freeze | grep "azure*" | xargs -n 1 pip uninstall -y

Show transaction history for a package

dnf history list <package>

Show information about specific transaction in history

More on ‘history’

dnf history info <transaction ID>

Undo last transaction

dnf history undo last

List SystemD timers

systemctl list-timers

Show execution of service tied to a timer of an identical name

journalctl -u name.timer
journalctl -u name.service

Copy remote directory to local system

scp -rCp <user>@<remote>:<remote path> <local path>
# OR (faster)
rsync -azvhP <user>@<remote>:<remote path> <local path>

Overwrite existing directory with contents from another

rsync -av --delete ~/new/ ~/old

Count number of installed kernels

$ sudo dnf list --installed kernel-core* | tail -n +2 | wc -l
9

Increase number of installed kernels in /etc/dnf/dnf.conf

...
installonly_limit=10
...

Pin specific kernel version

$ sudo dnf install python3-dnf-plugins-extras-versionlock
$ # List kernel packages
$ rpm -qa kernel
kernel-6.0.18-300.fc37.x86_64
kernel-6.1.7-200.fc37.x86_64
kernel-6.1.8-200.fc37.x86_64
$ Add pin
$ sudo dnf versionlock add kernel-6.0.18-300.fc37.x86_64
Last metadata expiration check: 3:51:11 ago on E 30 jaan  2023 15:47:21.
Adding versionlock on: kernel-0:6.0.18-300.fc37.*
$ # Remove pin
$ sudo dnf versionlock delete kernel-6.0.18-300.fc37.x86_64
...

Undo ad hoc changes made to a SystemD service

For example systemd-resolved.

$ systemctl revert systemd-resolved.service
Removed "/etc/systemd/system/systemd-resolved.service.d/override.conf".
Removed "/etc/systemd/system/systemd-resolved.service.d".
$ systemctl restart systemd-resolved.service

Back up a file using brace expansion

Equivalent to cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak. More on brace expansion.

cp /etc/ssh/sshd_config{,.bak}

The same can be applied for directories:

cp -aR public{,.bak}

Restore a backed up file

Equivalent to cp /etc/ssh/sshd_config.bak /etc/ssh/sshd_config.

cp /etc/ssh/sshd_config{.bak,}

Download older version of a kernel

Fedora Discussion here.

koji download-build --arch=x86_64 <kernel package name>

Read lines from a file into an array and execute in parallel

$ readarray -t items < items.txt
$ parallel -kj 20 echo {1} ::: "${items[@]}"

Verify SSL certificate against domain

openssl s_client -connect google.com:443 2> /dev/null | openssl x509 -noout -dates

Run programs with ‘systemd-run’ to leverage SystemD’s features

Documentation here.

$ systemd-run env
Running as unit: run-19945.service
$ journalctl -u run-19945.service
Sep 08 07:37:21 bupkis systemd[1]: Starting /usr/bin/env...
Sep 08 07:37:21 bupkis systemd[1]: Started /usr/bin/env.
Sep 08 07:37:21 bupkis env[19948]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Sep 08 07:37:21 bupkis env[19948]: LANG=en_US.UTF-8
Sep 08 07:37:21 bupkis env[19948]: BOOT_IMAGE=/vmlinuz-3.11.0-0.rc5.git6.2.fc20.x86_64

Working with ‘screen’

  • Start a session: screen
  • Start any long-running commands
  • Detach from session: Ctrl+A+D
  • List sessions: screen -ls
  • Attach to a running session: screen -r <session ID>