It happens from time to time that at some point it is hard to recall the familiar commands used before, meanwhile it is somewhat annoying and time consuming to find them again. As the old saying goes, “the palest ink is better than the best memory”. Instead of relying on muscle memory, this post serves as a growing memo for those handy commands.

Network Configuration

iproute2 is designed to be a more organized tool chain to replace net-tools which consists of /sbin/ifconfig and so on. At the core of iproute2 is /sbin/ip.

  • ip help, ip neigh help to view helping info at different levels
  • ip addr to show all addresses related info
  • ip -s link to view statistics (RX/TX pkts, dropped, bytes etc) of all interfaces
  • ip link set enp101s0f0 up or ifconfig enp101s0f0 up to bring the interface up
  • ip link set enp101s0f0 down: bring the interface down
  • ip link set enp101s0f0 promisc on: set the interface in promisc mode
  • ip addr add 10.1.1.3/8 dev enp101s0f1: assign ip to the device interface
  • ip route get 192.168.88.77: query the route for the specified ip
  • ip route add 10.1.1.0/24 dev enp101s0f1: add new route entry
  • ip route del default and ip route add default [via 192.168.1.1] dev enp101s0f1 to change the default gateway
  • ip neigh or ip n as a replacement of arp -a to view the arp table

  • ip n add 10.1.1.2 lladdr ec:0d:9a:6d:e1:6d dev enp101s0f0 nud perm as a replacement of arp -s 10.1.1.2 ec:0d:9a:6d:e1:6d to set a permanant arp entry

  • sudo ethtool enp101s0f1 prints the speed of the interface

  • sudo ethtool -i enp101s0f1 shows the driver version of the interface

  • traceroute <ip/url> is useful to gain knowledge of the network topology

iptables is an interface to operate on top of netfilter. iptables defines chains as when rules will be evaluated, including PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING, mirroring the corresponding hooks in netfilter (NF_IP_PRE_ROUTING, NF_IP_LOCAL_IN, NF_IP_FORWARD, NF_IP_LOCAL_OUT, NF_IP_POST_ROUTING). Example usage:

  • iptables [-t nat] -L [-v] lists current rules (-v to show verbose details) for the specified table. iptables table consists of filter (default), nat, and mangle.
  • iptables -D OUTPUT 1 -t nat deletes the first rule (1-indexing) rule from table nat, chain OUTPUT
  • iptables -t nat -A POSTROUTING -s 192.168.1.2 -o eno1 -j SNAT --to-source 158.130.4.222 configures a server A with internet connection to uses SNAT protocol to modify all outbound packets’ source ip (from priv to pub) so that packets from its cable connected server B (without internet access) can reach the internet
  • /etc/resolv.conf configures the DNS resolver, e.g., a line nameserver 192.168.1.1 in the file will ask the resolver to query for the name server with the specified IP.

Network Traffic Monitoring and Manipulation

  • tshark -s 64 -B 1024 -i enp101s0f1 > tmp: capture high-speed traffic (with 64B cap of packet size and 1024MB buffer) at the specified interface and redirect the outputs for future analysis
  • tshark -c 100 -i enp101s0f1: stop capture after specified number of pkt count, useful when only a small snapshot is needed
  • tshark -Y option allows an expressive specification of the filter, example filters:
    • tshark -Y 'icmp'
    • tshark -Y ip.addr == 10.0.0.1
    • tcp.dstport != 80
    • ip.addr == 8.8.8.8 and !(icmp or dns)
  • tshark -V option shows verbose pkt output (including all layers)
  • tshark -i enp5s0 -Tfields -E separator=, -e eth.dst -e eth.src prints only the dst and src address delimitered by “,”
  • tshark -i ens1 -Y "eth.type == 0x4321" -Tfields -E separator=, -e eth.dst -e eth.src similar to above, but filters the ether types being non 0x4321

As a prerequisite for installing DPDK and pktgen, Mellanox NIC Linux driver installation steps:

  • lspci -v | grep Mellanox verifies the existence of Mellanox network adapter
  • Install a compatible mellanox NIC driver version, e.g., MLNX_OFED_LINUX-4.9-0.1.7.0-ubuntu18.04-x86_64.tgz for ConnectX-4, Ubuntu 18.04 LTS, arch x86. md5sum <file> before the actual installation
  • Verification
    • /etc/init.d/openibd restart loads/restarts the driver
    • Double check the driver version installed using MST (Mellanox Software Tools): mst start; ofed_info -s
    • Check that Mellanox kernel modules are loaded: lsmod | grep mlx
    • ibdev2netdev -v maps the adapter port to the net device

Useful tips for .pcap manipulation:

  • editcap -F pcap -i 60 bigfile.pcap tmp.pcap splits bigfile.pcap into 60s chunks
  • tshark [-V] -r bigfile.pcap -Y "frame.time_relative <= 1.0" reads 1s chunk of bigfile.pcap
  • tcpdump -r bigfile.pcap -c <num> -w firstnum.pcap saves first packets
  • tshark -r bigfile.pcap -Y "frame.time_relative <= 60.0" -w tmp.pcap saves first 1min packets
  • tshark -r <original.pcap> -Y "eth.type == 0x0800" -w clean.pcap focuses on ether type 0x0800
  • tshark -r 1s_chunk.pcap -Y "ip.proto == 6 or ip.proto == 17" -w tcpudp_1s_chunk.pcap filters non TCP, UDP packets
  • capinfos -Trc 1s_chunk.pcap prints number of packets in the pcap
  • tcprewrite --dlt=enet --enet-dmac=00:11:22:33:44:55 --enet-smac=66:77:88:99:AA:BB --infile=<chunk_file> --outfile=output.pcap adds fake Ether header
  • tcpreplay -i enp101s0f0 --pps 450000 -T nano -K --preload-pcap -l 1 test.pcap replays test.pcap at 450000 pkt/s with loop number 1
  • tcpreplay -i enp101s0f0 -x 2.5 -T nano -K --preload-pcap -l 1 test.pcap replays test.pcap with 2.5 times of the capture speed (not that it shouldn’t coexist with –pps or –mbps)

iperf is a handy command line tool to launch TCP/UDP server/client pairs.

  • Configure the interface ip, arp/route table etc a priori for the client/server pair(s)
  • iperf -s -i 0.5 launches an iperf server with 0.5s report interval
  • iperf -c 10.1.1.3 -t 10 -P 16 launches an iperf client targeting server 10.1.1.3 transmitting 10s with 16 parallel client threads

File Management

  • find / -name "libpython3.4m.so.*" finds recursively the file with the specified name regex starting from the file system root
  • grep -r "bf_tm_ppg_hdl" . -I fines the files with the specified content, skipping binary files
  • tar -xvzf <resource tarball>.tar.gz is commonly used to unzip a tar.gz file
  • fdisk -l views all disk partitions
  • sudo dd if=<.iso file path> of=/dev/sdb1 creates a bootable image on a USB device partition (which appears to be file/dir in Linux) on Ubuntu.
  • mount /dev/sdb1 /mnt mounts the file system (dev/dsb1, can also be in the form of .iso etc) to the mount point /mnt (existing dir), umount /mnt is the reverse operation to cancel the mount point
  • mount [-l] lists all mounted file systems
  • sshfs <myuser>@<ip>:<remote dir path> <local dir (mount point)> mounts the remote file system

Shell Execution

  • sudo -E ./script.sh to launch the script with sudo while keeping the current env variables, one could compare the output among env, sudo env, and sudo -E env
  • env prints the env vars of the current env
  • env [<env var name>=<value>] [<command> [<args>]] enables the executation under a modified env where the specified env vars are replaced with the new values, the rest remains. E.g, the only diff of env USER=test env output versus original env is the USER env var value. We could also manipuate PATH env var to the path containing a specific version of a prog without modifying the global env (while the default PATH prioritizes the path of another version).

Compilation/Linkling

  • ldd <executable> prints the shared objects/libraries dependencies (with specific version) of the program, useful when debugging cannot open shared object file: No such file or directory errors. Note that often there is only one actual .so file libX.so.0.0 while others such as libX.so, libX.so.0 are just symlinks to libX.so.0.0, i.e., only the latest version is used by default.
  • LD_LIBRARY_PATH env var specifies a set of dirs to search for shared libraries by the dynamic linker during run time rather than compilation time. E.g., sudo env LD_LIBRARY_PATH=/path1:/path2 ./prog will enables the dynamic linker to search path1 and path2 for the shared libraries needed by the executable prog.

Hotkey

  • Ctrl-Alt-F1: the terminal prompt display
  • Ctrl-Alt-Del: reboot

Kernel

  • uname -r prints the current kernel version
  • lsmod prints the current kernel modules
  • sudo insmod kdrv.ko to insert a kernel module and rmmod is the opposite operation

dmesg is a powerful tool to debug by monitoring the kernel ring buffer:

  • dmesg -T prints the ring buffer with human readable time (rather than default time showing the time since the machine boot)
  • dmesg -x appends the message level (info, err, debug, warn…)
  • dmesg --level=err,warn prints only the message with the specified level
  • dmesg output is often combined with pipes, e.g., dmesg | head -30 prints the first 30 lines of dmesg original output, similarly dmesg | tail -20 prints the last 20 lines of the output, dmesg | grep -i dma prints messages with dma substring (case insensitive with -i option)

Authentication

  • ssh-keygen -t rsa [-b 4096] generates an RSA public/private key pair
  • ssh-keygen -l -f ~/.ssh/leoyu.pub prints the number of bits of the specified key

RSA is similar to Diffie–Hellman key exchange in that they both leverage the property of the one-way function. RSA secretly picks 2 distinct prime numbers p, q and releases the modulus \(n=pq\). The public exponent e and private exponent d are chosen such that \(ed\equiv 1 \mod \phi(n)\) where \(\phi(n)=N-p-q+1=(q-1)(p-1)\) denotes the Euler’s totient function. The magic here is that \((m^{e})^{d}\mod n = m^{k\times \phi(n)+1}\mod n=(m^{\phi(n)})^{k}\times m\mod n\) and with Euler’s theorem, \((m^{e})^{d}\equiv m \mod n\). One could also show such equivalence using Fermat’s little theorem. This indicates that by applying both encryption and decryption (order doesn’t matter), we would recover the original message \(m\).

  • ssh -i <privkey path> <user>@<server ip> to ssh with the specified private key when it is absent in ssh-agent
  • ssh-add -l prints the currently added private keys, ssh-add ~/.ssh/id_rsa adds the specified private key to ssh-agent (needs appropriate access permissions, e.d., 600), and ssh-add -d ~/.ssh/id_rsa deletes an added identity

  • passwd [user name] to update the password for the current/specified user

  • openssl genrsa [-aes128|-aes192|-aes256] -out private.pem 512 generates a 1024b private key (with optional AES encryption in CBC mode of PEM output)
  • openssl rsa -in private.pem -pubout > public.pem derives the corresponding public key for the specific private key
  • openssl rsa -in private.pem -text -noout displays the modulus and the private exponent
  • openssl dgst -sha1 -sign private.pem -out sha1.sign msg.txt signs the specified file using SHA-1 digest algorithm using PKCS1v1.5, one could read the signature hexdump sha1.sign
  • openssl dgst -sha1 -verify public.pem -signature sha1.sign msg.txt verifies the signature

Signature generation workflow under the hood:

  • A cryptographic-hash function generates a digest (160 bit hash with SHA-1) for the message alice says hello to bob. E.g., run shasum -a 1 msg.txt on MAC or sha1sum msg.txt on Linux to generates SHA-1 digest f669524891e50a82cb6e355af792b0a8f6351241 for the message (cut -d ' ' -f1).
  • The digest is padded to RSA key size k bits w.r.t. a padding scheme (why? because it is deterministic otherwise), e.g., PKCS1v1.5 or a latest version PKCS1v2.2. In PKCS1v1.5, 00||01||PS||00||T||MD where 01 is the mode for digital signature, PS is a variable number of FF bytes (\(k/8-38\)), T is a a string of magic ASN.1 (nothing but a specification to describe data structure, example below) bytes specifying which digest mechnism to use, for SHA-1, it is 3021300906052b0e03021a05000414 (see P46 of PKCS1v2.2), and MD is the previously generated digest. 
    DigestInfo ::= SEQUENCE {
digestAlgorithm DigestAlgorithmIdentifier,
digest Digest }
  • To summarize, the padded message in the example above is 
    0001ffffffffffffffffffffffffffffffffffffffffffffffffffff003021300906052b0e03021a05000414f669524891e50a82cb6e355af792b0a8f6351241
  • Now we can sign the padded msg via \(c=(PKCS1V1\_5(MD))^{d}\mod n\) leveraging Python’s big number arithmetic. One could verify that the resulting sign should be identical to sha1.sign using openssl direcly. 
    md_padded = int('0001ffffffffffffffffffffffffffffffffffffffffffffffffffff003021300906052b0e03021a05000414f669524891e50a82cb6e355af792b0a8f6351241', 16)
n = int('00bb78017b6de556689dfd97f737003ab49fc5e463a73f02d6adf6cb725aebbe8db9f49fb45b5d56b9d9fe68309ccead978e33f8f03a43e4e40686a7078ac9db57', 16)
d = int('0090ef9ad5be850c651da0ae18a7213a41e334affa8b971faeb21897f9f569458c223adf2bb8bb9a23aa7522d95495e50b2f87095e5b9c461cc7d5764c8dcbbde1', 16)
sign = hex(pow(md_padded, d, n))[2:] # Remove the 0x prefix
  • The verification of a signature is similar. \(MD=PKCS1V1\_5^{-1}((c)^{e}\mod n)\)
  • Note that though the scheme under PKCS1V1_5 is IND-CPA (semantically secure), it turns out to be vulnerable to Bleichenbacher attacks. PKCS1V2_2 addresses the issue using a more sophisticated padding scheme for both signature/encryption to achieve IND-CCA.

Version Control

Git is probably the most popular version control systems (VCS) among others: Subversion (SVN) is a centralized VCS and Mercurial (Hg) is another distributed VCS.

  • git reset --hard <commit hash> points HEAD to a history commit AND rolls back the files
  • To mirror an existing Git repo A to another Git repo B, deletes the existing remote via git remote rm origin and adds a new remote via git remote add origin https://github.com/liangcheng-yu/another.git. Then overwrites repo B with git push -f --set-upstream origin master.

Job Monitoring

  • top is commonly used to monitor currently dominant jobs and their usages. Press c to show full command for each job and Y to pause at current snapshot
  • ps -aux | less displays all currently running jobs one page at a time

Cables and Wiring

  • Serial cable connects the serial/console ports of two devices communicating via serial communication protocol. minicom is used to configure the baud, parity, stop bit and so on.
  • Breakout/fanout cables are often used to split the port bandwidth, e.g., a QSFP28-4SFP28 breakout cable split a 100G port of a switch to 4x25G server ports.
  • Ethernet cables that run inside walls and crawl spaces are different from those meant to plugged in/out the computer repeatedly. The former is fragile to rewire and ideally settled once while the latter is more flexible to bent.
  • Fixed Optical Attenuator, Variable Optical Attenuator
  • Video Graphics Array (VGA) cable
  • Transceiver: Fibre Channel and Ethernet Transceiver both have their own targeting use cases
  • Common form factors (a set of specifications (size, shape…) during hardware design, in this context, for transceivers): 1G SFP, 10G SFP+, 25G SFP28, 40G QSFP+, 50G QSFP28, 100G QSFP28, 200G QSFP56, 400G QSFP-DD.

    A nice FiberStore (FS.COM) post on the differences and backward compatibility of common form factors. E.g., SFP28 supports only one channel with 25 Gbit/s, while QSFP28 supports 4 separate lanes, each of 25 Gbps. In 100G networks, one could use QSFP28 to SFP28 breakout cables to connect a 100G switch with a 25G switch.