Skip to content

shell-one-liners.md

Latest commit

 

History

History
527 lines (365 loc) · 13.2 KB

shell-one-liners.md

File metadata and controls

527 lines (365 loc) · 13.2 KB
Raw

Shell One Liners

Because who doesn't love shell tricks.

For far more serious tricks see the DevOps-Bash-tools repo.

System & Shell

Show CPU Cores

nproc

Show your $PATH entries one per line

echo $PATH | tr ':' '\n'

Log All Commands in a Shell Session

script logfile.txt

Log Script Output to a File

You can wrap part or all of your script in braces and use standard output redirection to a logfile:

{
    echo "capturing this output"
} > output.log 2>&1

To capture all stdout from this point down in the script to a logfile eg. output.log, as well as still print it to the terminal:

exec > >(tee output.log)

Sometimes you want the stderr to go the terminal as that's where you log progress of what the script is doing while collecting the output data.

To have stderr also be logged into the output.log file:

exec > >(tee output.log) 2>&1

This is using the Bash trick of having a process as an input or output file descriptor, similar to Use Process Outputs Like Files.

List Your 100 Most Used Bash Commands

If your shell history is vanilla without timestamps:

history | awk '{ a[$2]++ } END { for(i in a) { print a[i] " " i } }' | sort -rn | head -n 100

If you're using timestamped shell history:

history | awk '{ a[$4]++ } END { for(i in a) { print a[i] " " i } }' | sort -rn | head -n 100

Processes

Find disowned processes owned by the init PID 1

 ps -ef | awk '$3 == 1 {print}'

Use Process Outputs Like Files

Access command outputs like files without using temp files.

The <(...) Bash syntax returns a temporary process file descriptor.

some_command_expecting_a_file <(some_command_printing_to_stdout)

For example, compare two unsorted files without using temp files:

diff <(sort "$file1") <(sort "$file2")

Dates

Get Current Epoch

Seconds since Unix birth 1st Jan 1970:

date '+%s'

Convert Epoch Seconds to Human Readable Date

date -d @1741942727

Files & Strings

Generate ASCII Art

brew install figlet
figlet "Hari Sekhon"
  _   _            _   ____       _    _
 | | | | __ _ _ __(_) / ___|  ___| | _| |__   ___  _ __
 | |_| |/ _` | '__| | \___ \ / _ \ |/ / '_ \ / _ \| '_ \
 |  _  | (_| | |  | |  ___) |  __/   <| | | | (_) | | | |
 |_| |_|\__,_|_|  |_| |____/ \___|_|\_\_| |_|\___/|_| |_|

Number Lines

cat -n
less -N
nl

Copy to Both Clipboard and Stdout simultaneously

tee to both a command to copy to clipboard as well as stdout.

Use /dev/stdout for further pipeline processing, not /dev/tty, as the latter outputs directly to the terminal.

The copy_to_clipboard.sh script from DevOps-Bash-tools works on both Linux and Mac:

echo test | tee >("copy_to_clipboard.sh") /dev/stdout

Squeeze Out Multiple Blank Lines

Useful to remove multiple blank lines between paragraphs in text replacements like shorten_text_selection.scpt.

cat -s

Reverse a String

echo "testing" | rev

Reverse the lines of a file

More portable than tac:

tail -r file.txt

Shuffle Lines of a File

shuf file.txt

Split Big File into 10MB chunks

split -b 10M bigfile split_

Show Files Open by a Process

lsof -p "$(pidof bash)"

Generate a Random Password

brew install pwgen

Generates a bunch of passwords to choose from

pwgen

Or generate a single one of length 20 chars and avoid any ambiguous chars:

pwgen -1 -s -B 20

Base64 Secrets to avoid dodgy characters

Useful in things like GitHub Actions CI/CD for safer character handling.

base64

use --decode switch instead of -d for better portability between your Mac and Linux:

base64 --decode

Deduplicate Lines Using Awk

If you have an input such as <id> <content> and want to deduplicate lines with the same ID by comparing the first field, this is simple and powerful:

awk '!seen[$1]++'

You could of course deduplicate entire lines by just using $0 as the associative array index:

awk '!seen[$0]++'

This uses the trick of populating an associative array using the field as the index and an ++ incrementer to just create a value (1) to signal existance.

This follows standard computer science post-increment behaviour such that the first time it does this it returns nothing since it didn't already exist and so the leading negation will allow the first instance through and print the line, but subsequent instances where the value used for the index, $1 for the ID field or the $0 for the entire line, actually returns an existing value from the last time that index value was seen and incremented into existence, it'll match the negation and suppress those lines.

Find Lines in a File present in Other Files

I used this to find Spotify tracks that are in one of my blacklists as I download all my playlists to Git revision controlled backups and use scripts to remove songs I've already checked (HariSekhon/Spotify-Playlists).

  • -F - for fixed string
  • -x - to ensure we match whole lines
  • -h - to not return the prefix of "$fileN: " before each line printed as we just want the line itself
  • -f - take the lines as patterns from the first file and look for them in all subsequent file arguments
grep -Fxhf "$patterns_file" "$file1" "$file2" ...

Pipe to sort -u to deduplicate if something is found in more than one file.

Find Lines in a File not present in Other Files

Same as above just grep -v it:

grep -Fvxhf "$patterns_file" "$file1" "$file2" ...

Example:

I use this to find tracks in my Spotify playlists that are not in Spotify's Like Songs so I can quickly add them programmatically.

The spotify/<files> paths contain the downloaded track URIs, while the top level contains the Artist - Track human readable downloads.

Find the URIs in a playlist that are not in the Liked Songs, which we use as the first arg, the patterns file:

grep -Fvxhf "spotify/Liked Songs" "spotify/$playlist"

and then pipe it to one of my many spotify scripts that can be found in DevOps-Bash-tools to programmatically 'Like' all of the straggers found.

First exclude local tracks which can't be Liked, and then pipe it to spotify_uri_to_name.sh to see the track names:

grep -Fvxhf "spotify/Liked Songs" "spotify/Smooth Hip-Hop 😎" |
grep -v "^spotify:local:" |
spotify_uri_to_name.sh

Once you've reviewed the songs to be Liked, pipe it to spotify_set_tracks_uri_to_liked.sh instead of spotify_uri_to_name.sh, to actually programmatically Like them:

grep -Fvxhf "spotify/Liked Songs" "spotify/Smooth Hip-Hop 😎" |
grep -v "^spotify:local:" |
spotify_set_tracks_uri_to_liked.sh

Removing Files with Difficult Special Characters

Sometimes you're coding and generate filenames with awful characters such as newlines, unicodes or emojis that can be either difficult or downright dangerous to try to remove on the shell command line.

In this case, cheat - outsource the deletion to an interactive tool like your GUI File Browser such as macOS Finder or on the command line using an interactive tool like Midnight Commander.

Install Midnight Commander on macOS using Homebrew:

brew install mc

Start Midnight Commander:

mc

Scroll down to the file or directory and then hit Esc - 8 to prompt deletion.

Esc - fn - 0 to exit on macOS.

Network

Get Your Public IP Address

These tricks use publicly available services to return what your actual public IP address is after all NAT translation ie. what your IP is actually seen as by other computers on the internet.

curl ifconfig.co
curl checkip.amazonaws.com

or using just DNS:

dig +short myip.opendns.com @resolver1.opendns.com

Websites that return the IP without a trailing newline:

curl ifconfig.me
curl https://ipinfo.io
curl ident.me

Programmatically useful with more details returned in JSON:

curl ifconfig.co/json

List Open TCP/UDP Ports

Show your local listening ports:

netstat -lntpu

Check if a Port is Open

nc -zv localhost 22
localhost [127.0.0.1] 22 (ssh) open

Wait and Make Audible Bell Sound when a Host is Online

Checks every 3 seconds, makes an audible bell, and exits when a host comes online:

ping -i 3 -o -a "$host"

Quickly Serve Local Files over HTTP

python3 -m http.server 8000

Top for Network Processes

:octocat: raboof/nethogs

brew install nethogs
sudo nethogs

Disk

Find Biggest Files Taking Up Space

This searches only the current directory and only on the current partition, not sub-mount-points.

du -max . | sort -k1n | tail -n 1000

See also for a GUI alternative:

Create a Ram Disk

mkdir /tmp/ramdisk &&
mount -t tmpfs tmpfs /tmp/ramdisk -o size=1024m

Mac

Use GNU CoreUtils

For cross-platform portability of your commands and scripts since GNU CoreUtils have better features than their BSD counterparts, and less bugs in advanced usage in my experience.

brew install coreutils

This will install GNU Core Utils prefixed with g to not clash with the native BSD counterparts.

Mask BSD Core Utils with GNU Core Utils

You can wrap them in functions (I do this in scripts and libraries eg. DevOps-Bash-tools:

grep(){
  command ggrep "$@"
}

sed(){
  command sed "$@"
}

Now when the script calls grep it gets the better functional version for cross-platform compatibility between Mac and Linux.

More Resources