unix

Table of ContentsClose

1. Common usage
2. one-liner dumps
3. More than a command
- 3.1. Flash drives

1. Common usage

1.1. Redirection

> is the redirection operator
& is an indicator that the following word is a fd, not a filename.
- Eg. >1: redirecting to a file named 1, >&1: redirecting to fd:1

1.1.1. stdout and stderr

POSIX standard
- redirect stdout
- redirect stderr to stdout
- command >/dev/null 2>&1
Bash shorthand (redirect both stdout and stderr)
- command &> /dev/null

1.2. Heredoc

This is bash only, fish does not support heredocs(<<)
<<< is same but limited by one line

# redirected to file
cat <<POOP>somefile.txt
something
something else
POOP
# redirected to stdout
cat <<POOP
something
something else
POOP

1.3. Bash

bash runs all the commands in the script in the same process group

1.4. xargs

# file creation
cat list.txt #a b c d
xargs -t touch < list.txt # a b c d list.txt
echo "x y z" | xargs touch # a b c d list.txt x y z

# replace string
seq 10 | xargs -I X echo xXx
# x1x
# x2x
# x3x

seq 9 | xargs -L 3
# 1 2 3
# 4 5 6
# 7 8 9

2. one-liner dumps

# from local to server
rsync -avzhP -e "ssh -i path_to_key" /from/dir/ username@host:/to/dir/

# grep and replace
$ grep -rl assets | xargs sed -i 's/assets/\/img\//g'

# list duplicates using jq
cat myson.json | jq '[.data[].url]| group_by(.) | map(select(length>1))'

# remove a field from each object in an json array
cat data.json | jq 'map(del(.property_name))'

# Poor man's stress test by @bwplokta
# Replaces every newline with x then `head` for 4GB(4*10^9bytes)
# then feed it to `grep`, grep will slowly consume 4GB of memory.
yes | tr \\n x | head -c $BYTES | grep showmedamoney

3. More than a command

3.1. Flash drives

3.1.1. Formatting drive

dd if=/dev/zero of=/dev/sdX status=progress # clear out the disk
fdisk /dev/sdb
mkfs.vfat /dev/sdb1 # or mkfs -t vfat /dev/sdb1

Putting a FS on a partition == "Making a filesystem" (FS is what gets mounted)

3.1.2. Data recovery

3.1.3. Other notes

Sometimes usb devices may not show up in lsblk after kernel upgrade even if dmesg is alright and lsusb detects things. This is normal. Just reboot.