Automating frequent tasks

More about redirecting input and output

A program running under the shell can have several files open to it simultaneously for reading and writing. They are identified by their file descriptors, numbers used by the system to associate a file with an input or output stream. The system treats each open file as a stream of characters that flow sequentially, from start to finish. The streams associated with any program are the standard input, represented by file descriptor 0, the standard output (file descriptor 1) and the standard error (file descriptor 2).

Basic shell syntax

The basic shell syntax for redirecting input and output is as follows:

<file: Use file as a source of standard input.
<nfile: Read file as a source of input to file descriptor n.
>file: Write standard output to file.
n>file: Write the output from file descriptor n to file.

In the following example, the file called thing does not exist:

   $ cat thing
   cat: cannot open thing: No such file or directory
   $ cat thing 2> /dev/null
   $

This effect is particularly useful when appended to a command that generates copious but unwanted error messages; it sends the output from file descriptor 2 (the standard error) to /dev/null, the ``bit bucket'' or ``black hole'' device. (/dev/null is also known as the null device; if you send data to it, it absorbs it silently, and if you read from it all you get is a null character.)

Other useful fragments are:

>&2: appended to an echo, sends the output to the standard error
2>&1: merges the standard error with the standard output

Note that when the ``>'' symbol is employed, the file it is directing output to is either created or, if it already exists, is erased and replaced. This is known as ``clobbering'' a file. (The system knows better than to destroy terminal or tape special device files this way: the tape or screen controlled by the device is overwritten, but the device file itself in /dev is not affected.)

To append output to the end of an existing file, use the ``>>'' notation instead.

If you want to permanently prevent the Korn shell from destroying an existing file when you use the ``>'' redirection operator, adjust the shell parameter noclobber by issuing the command set -o noclobber. If the shell finds that a file it is writing to already exists, it will issue an error message and refuse to overwrite it, as follows:

   $ cat aaa > bbb
   ksh: bbb: file already exists

Once noclobber is set, you have to redirect using the override command, >! (instead of >) if you want to overwrite it.

The << operator has a special meaning: it is used to tell the shell to read its standard input from the current script. For example, if you have a shell script containing the line:

   <<terminating_string
    :
    :

Everything from that line down, until it encounters a line with just ``terminating_string'' on it will be taken as a here document, a file which is treated as the standard input. So, to send a multiline message to the screen, instead of using print or echo you could embed a help message in your script:

_help() { cat <<%%

Readability Analysis Program

A shell/awk demo to determine the readability grade of texts

Either run rap with no options for full menu-driven activity, or use the following flags:

-[h|H] prints this help -l cause output to be logged to a file -f file enter the name of the file to check -b run in batch mode (no menus)

%% exit 1 }

This defines a function called _help within a shell script. When the script subsequently encounters the command _help it will cat the text between two sets of ``%%'' symbols to the standard output, then exit.

Scripts running under the shell may have many file descriptors in use simultaneously. Some programs may not be able to deal with reading and writing lots of redirected file descriptors: other programs expect to read a filename on their command line, rather than look for redirected input.

To get round this, you can use the special files /dev/stdin, /dev/stdout, and /dev/stderr; see ``Forcing a program to read standard input and output'' for an example of this.

The following example shows an instance of extracting streams of information from one file and placing them in two different output files using only one pipeline, as follows:

2>second_field; cat myfile | awk '{ print $2 > "/dev/stderr"; print $1 }' | sort

The first command on this line attaches the standard error to a file (in this instance second_field). The input file myfile is then piped into an awk program. The awk program prints the second field of every line to /dev/stderr, the standard error, and prints the first field of every line to the standard output. Because the standard error has been redirected, the second field of each line ends up in second_field, while the first fields are sorted and presented on the standard output.