From LQWiki

sed (stream editor) is an editor used, not interactively on text files (like vi or emacs), but on streams. This allows it to transform text input from a pipe or the command line or a file (if it is piped to sed or given as an argument to sed).

sed supports regular expressions which gives it great control over what it can do to the input stream. Common applications of sed include parsing log files, replacing specific words/typos in a stream, and reading CSV files.

Substitution

The Substitute command is quite powerful. In its basic form, it consists of 's/old/new/g', which will replace all occurances of "old" with "new". Of course, the basic form can be changed significantly.

Regular Expressions: Regular Expressions allow replacement of patterns instead of strings. sed uses the standard [-]*^ regex syntax. For example, 's/[0-9]*//g' will remove all numbers from the input.

Remembering the input: sed can remember parts of the input pattern, and replace that section in the output file. This is done through the &, \(, \), and \1-9 commands.

The & command, when used in the replacement pattern, is replaced with the input string. For example 's/[^ ]*/&&/g' will double the input. The \( \) commands allow for more precise work; any text between the first \( occurance and the first \) occurance can be placed into the output with \1. The second \( \) pair can be recalled with \2, and so on, up to \9. For example, 's/\([0-9]\)\([0-9]*\)/\2\1/g' will take numbers, and put the first digit of the input on the end of the output.

Multiple Substitutions: Normally, sed can only make one substitution at a time; sed 's/old1/new1/g' 's/old2/new2' will not work. However, the -e option allows multiple commands. Place a "-e" before all commands: sed -e 's/old1/new1/g' -e 's/old2/new2/g'

Examples

One of the most common uses of sed is to make a quick substitution in a file or data stream. This is done with the following:

$  command | sed 's/regexp/replacement/g'

Here, command generates the input data, regexp is the text to search for and remove, and replacement is the new text to insert. The g suffix causes sed to make more than one replacement per input line as is usually intended.

sed is commonly also used as a pre-processing step before handing off to another program. A specific example of this would be to strip out all HTML tags from a file before passing the file to grep for the word internet.

 $ sed 's/<.*>//g' foobar.html | grep internet

sed also works for text editing tasks. For example, the command

 $ sed -e 's/foo \([^ ]*\)/foo("\1")/g' -e 's/bar \([0-9]*\) \([^ ]*\)/bar(\1, "\2")/' < inputfile.txt > outputfile.txt

will search the file inputfile.txt for all strings of the form foo something and replace them with foo("something"). It will also search for strings of the form bar 543543 something and replace them with bar(543543, "something"). It will then output the resulting file to outputfile.txt.

Examining the above command in more detail: the -e flags specify a new expression, allowing more than one replacement rule in the same command. The escaped-parentheses \( and \) tell sed to remember the string that was enclosed. The \1 and \2 commands tell sed to insert a remembered string; numbers 1-9 all work as expected. The [0-9]* and similar commands are just Regular Expressions.

Sed scripts

If you want to use a lot of commands you can save them as a sed_script.txt:

# this is a sed script
# usage: $ sed -f sed_script.txt textfile.txt > new_textfile.txt

# start replacing all things that must be replaced
# replace / with _
s/\//_/g 

Running the script on textfile.txt:

$ sed -f sed_script.txt textfile.txt > new_textfile.txt

Another option is to save it as an executable script script.sh:

#!/usr/bin/sed -f
# save your commands in this script.sh; chmod u+x script.sh
# call it like so: $ ./script.sh textfile.txt > new_textfile.txt

# start replacing all things that must be replaced
# replace / with _
s/\//_/g

To execute the script on your textfile.txt:

$ chmod u+x script.sh
$ ./script.sh textfile.txt > new_textfile.txt

See also

• RegEx
• sed man page (man.linuxquestions.org)
• GNU sed online manual (www.gnu.org)
• Handy one-liners for sed (www.student.northpark.edu)
• Sed, the Stream Editor by example (www.itjungle.com)
• An introduction to sed, with an example for sed -f and executable scripts (muddcs.cs.hmc.edu)