Piping
Piping is where the standard output of one process is sent to the standard input of another. In essence, one program's output is modified or used in some way by another. In most shells, this is done by putting the pipe character | between commands.
This is treated as a descriptive metaphor for plumbing in general. Thus the commands pipe, tee, and the terms pipe (the connection between the processes,) fifo (a named pipe,) pipeline which refers to the processes linked together by pipes.
Shell usage
Piping allows the efficient text processing capabilities Unix and Unix-like system were originally designed for, amongst other capabilities.
One example is to use output from dmesg as input for grep, to list only the lines containing a given string:
$ dmesg | grep EXT3 EXT3-fs warning: maximal mount count reached, running e2fsck is recommended EXT3 FS on hda5, internal journal EXT3-fs: mounted filesystem with ordered data mode. $
Maybe you'd like to find all the line numbers where Gimli is mentioned in the Lord of the Rings. The nl command will print line numbers on things that are piped in to it. We can then grep for Gimli to sort out the lines we're looking for. There are quite a lot of lines however, so we can use more (or less) to show one screenful at a time (hit space for more):
$ cat Lord_of_the_rings.txt | nl | grep Gimli | more 9592 There was a younger dwarf at Glóin's side: his son Gimli. Beside 11064 shall be for the Elves; and Gimli son of Glóin for the Dwarves. They 11220 Gimli the dwarf alone wore openly a short shirt of steel-rings, for 11276 the road darkens,' said Gimli. 11279 'Yet sworn word may strengthen quaking heart,' said Gimli. --More--
Maybe you like Gimli enough to make him the hero of the story by replacing "Frodo" with "Gimli" using sed:
$ cat Lord_of_the_rings.txt | sed -e 's/Frodo/Gimli/g' | more
(...)
The eldest of these, and Bilbo's favourite, was young Gimli Baggins.
When Bilbo was ninety-nine, he adopted Gimli as his heir, and brought
him to live at Bag End; and the hopes of the Sackville-Bagginses were
finally dashed.
--More--
Useful commands for piping:
- cat to dump files
- nl to number lines
- wc to count words
- grep to match lines
- ppt to turn the input into ticker tape
- netpbm to work with images
- sed and awk for processing text
- xargs for using input as arguments to a command rather than stdin.
Programming details
The pipe that most shells use to send data between programs is a one-way no name pipe that can be created with pipe() which have the following prototype:
int pipe(int p[2]);
This creates a pair of file descriptors where p[0] is for reading and p[1] is for writing. Typically you let the transmitting process write data to p[1] and the receiving process read data from p[0]. However, programs typically read data from stdin and write to stdout, STDIN_FILENO and STDOUT_FILENO is the standard file descriptors for these. In order to make the processes use the pipe rather than stdin and stdout you can use dup2. For example:
dup2(p[0], STDIN_FILENO);
This makes p[0] 'look' like stdin from the process point of view.
Another way to program pipes is with the popen() function. It creates a new stream that sends or receives its data to/from another process:
FILE *popen(char *command, char *readwrite);
The command is opened for reading or writing, depending on the string readwrite, which must be either "r" or "w". The pipe should be closed with pclose().