GNU and other utilities

The real power in Unix-style utilities is that each is specialized in a task and they can be combined in a pipeline to accomplish more complex tasks (the divide and conquer approach).

Rather than going very deeply into all features of these utilities, this page gives you a flavor of what they can do. It's basically the equivalent of showing that a hammer, screwdriver and pliers exist and after browsing this you should be able to imagine which tool is best suited for a certain task.

It can show the contents of a file:

cat readme.txt

It can also be used to concatenate the contents of two files and place it in a new combined file:

cat my_file1 my_file2 > combined_file

Can be used to generate text. This can be useful in a shell script to show text.

echo Hello
Hello

With head and tail you can view the first and last lines (respectively) of the input. By default 10 lines are shown, but with argument -n you can change the amount of lines. Example to show the first 15 lines of a file:

head -n 15 my_file

A useful argument of tail is -f to follow a growing file (e.g. a log file). It will start by showing the last 10 lines and then show new lines as soon as they are added to the file:

tail -f my_log_file

Allows simultaneously showing text and redirecting it to a file:

echo hi | tee my_file
hi
cat my_file
hi

Can count words or lines. Example to count the amount of lines in a file:

cat my_file | wc -l

Grep can find a text pattern. This is the most basic example to find a string (the text “foo” in this example) in a file. Option -i makes it case insensitive, such that e.g. “Foo” or “FOO” will also be a match.

grep -i foo file.txt

Grep only outputs the lines where the pattern has been found. An example of using grep to find a pattern in the output of another command:

ps aux | grep foo

Grep stands for General Regular Expression Parser. With regular expressions (regex or regexp for short)¹⁾ you can create a more complicated search patterns. You can for example look for:

grep "ab.xy" file.txt

The dot means any character is allowed there. This pattern therefore will match e.g. “abcxy”, but also “abzxy”.

These things are possible in a regex for grep:

Example of a regexp can will match e.g. “abbx” and “ayx” as long as there is no a, b or c after it:

grep "a.*x[^abc]" file.txt

If you use argument -E for grep, then you enable these extended regexps:

Sort alphabetizes multiple lines of text. Example:

cat my_file
aa
bb
aa
cc
sort my_file
aa
aa
bb
cc

This is often used in combination with uniq, which removes duplicates from sorted lines:

sort my_file | uniq
aa
bb
cc

This utility can cut a specific section from each line. For example to keep only characters 2 to 5:

echo "abcdefghijklop" | cut -c 2-5
bcde

It can also cut by column number. You need to indicate what character is the delimiter between the columns (in this case a space):

echo "ab cd ef gh ij kl" | cut -d " " -f 2-3
cd ef

Stands for: translate (or delete) characters. Can be used to replace certain characters with other characters. Example to replace all b's with c's:

echo "aabbbb" | tr 'b' 'c'
aacccc

Example to replace all upper-case letters with lower-case ones:

echo "AaBcCc" | tr 'A-Z' 'a-z'
aabbcc

Sed stands for stream editor. It can replace certain text. A simple search and replace is done in this example. The g at the end means global and causes all occurrences in a line to be replaced. The default behavior without the g is to only replace the first occurrence.

echo "ab cd old gh old" | sed -e "s/old/new/g"
ab cd new gh new

Instead of a fixed string (like “old” above) you can also use regular expressions with sed.

To replace text in a file you can use the -i (“in place”) argument to allow sed to modify the file itself:

sed -i -e 's/old/new/g' my_file

Awk is more than just a utility. It's an entire programming language, but you can also create one-liners for the command line. Like cut you can also use awk to handle text that has columns. It also supports regular expressions.

Example to extract a particular column. The columns are indicated with $1, $2, etc. Argument -F (for field separator) can be used to indicate the delimiter between the columns. By default is assumes spaces.

echo "ab;cd;ef" | awk -F ';' '{print $2}'
cd

Example to extract only lines with a certain pattern and then only the 2nd column from that line. The ifconfig command shows information about your network connections, the line with pattern “inet ” is about the IPv4 address and the 2nd column is the IP address itself.

/sbin/ifconfig eth0 | awk '/inet / {print $2}'
10.33.15.88

Can be used to create a nicely formatted table in which each column is wide enough for the widest entry. You can indicate the delimiter between the columns with argument -s.

cat my_file
aa,bbbb,cc,ddddd
eeee,ff,ggggg,h
column -t -s ',' my_file
aa    bbbb  cc     ddddd
eeee  ff    ggggg  h

Tells you where a binary is located. Example to show where du can be found:

which du
/bin/du

Can tell you where a certain file (of any type) is located on your file system. It does not search for the file live, but checks a database with all filenames that is updated regularly. Not all distributions have locate installed by default.

locate my_file
/home/jarkko/my_file

Find searches your disk live. You need to indicate the starting directory for the search and what you are looking for. Example to find all files with “foo” in their name, starting from the current directory, ignoring the difference between lower and upper case (argument -i means case insensitive) in the filename:

find . -iname "*foo*"

You can specify the type to search for with argument -type. Most commonly used are f for regular files, l for symlinks and d for directories. Example to create a listing of all subdirectories underneath the current directory:

find . -type d

Argument -exec can be used to run some command for every search result. Where it says '{}' the search result will be filled in and ';' indicates the end of the command. Example to look for all files with “foo” in their name and then check for each whether they contain the text “bar”:

find . -iname "*foo*" -exec grep -l bar '{}' ';'

While the -exec argument for find can execute a specific command separately for each found file, the xargs utility can pass a list of files to a command that is run only once. That is considerably less overhead and faster. Example:

find . -iname "*foo*" | xargs grep -l bar

Diff shows what is different between two files:

diff my_file1 my_file2

If you specify two directories, the differences are shown between each pair of identically named files in the directories. There are many editors and specialized tools that can show the output of diff in a more convenient way.

The output of diff can be used by patch to apply those differences and transform it into the other file that was compared with diff.

patch original_file patch_file

It changes the access time (meaning the last time the file was opened) of a file. In practice I use this utility for something else entirely. If the file does not exist yet, it creates it. So it can be used as a quick way to create a new empty file:

touch my_file

Identify the type of some file (e.g. binary, shared library or font). Can be useful if you have no idea what kind of contents it has. Example:

file 5x8.pcf

Can be used to make a one-on-one copy of a partition and put it in a file (e.g. as a back-up):

dd if=/dev/sda1 of=partition_copy.img

Can download a file from a website or (S)FTP server. It shows the progress while downloading.

wget https://some.server.com/remote_file

Can download a file from a website or (S)FTP server:

curl https://some.server.com/remote_file -o local_file_to_save

This tool is often used in instructions how to download some open-source project. For example:

curl https://some.server.com/install_script | bash

If not saving to a file curl outputs the downloaded file directly, which allows piping it to a shell to execute the script directly.

To synchronize what can be found in a directory on a remote Linux machine (that you can log into) towards a local directory (or vice versa). Synchronizing means that it will only transfer the differences and not all files. This can be useful for back-up purposes. Example:

rsync -a remote_user@remote_hostname:/some/remote/path /some/local/path

Argument -a means archive mode. That includes fetching all directories recursively and preserve things like the owners and permissions of the files.

If you can log into a Linux machine with SSH, then you can you use scp to copy files to/from that machine. Example to download a file from the remote Linux machine:

scp remote_user@remote_hostname:/some/remote/path

You can also use any graphical program that supports the SSH protocol, such as WinSCP for Windows.