I Thought I Knew sed, I Didn't!

amirhosein 7 min read 1366 words

If you’ve worked with shell scripts, Dockerfiles, or CI pipelines, you’ve probably come across these:

sed 's/foo/bar/g'
sed 's#foo#bar#g'
sed 's|foo|bar|g'

They all do exactly the same thing. The only difference is the delimiter.

But why does sed allow different delimiters? And what else can you do with it besides search and replace?

Let’s take a closer look at addresses, commands, flags, and command-line options.

Examples are tested with GNU sed 4.9. Features marked (GNU) aren’t supported by macOS/BSD sed. See the portability notes at the end.

Anatomy of a sed instruction

Every sed instruction follows this structure:

[address[,address]][!]command[arguments/flags]

For example:

sed '/^#/!s#foo#bar#g' file

Part	Meaning
`/^#/`	Address — which lines to process
`!`	Negate — select lines that don’t match
`s`	Command — substitute
`#foo#bar#`	Arguments — pattern and replacement, using `#` as delimiter
`g`	Flag — replace all matches on the line

Delimiters: `/` is just a convention

The character immediately after s becomes the delimiter. POSIX allows any single character except backslash and newline.

sed 's/foo/bar/g'
sed 's#foo#bar#g'
sed 's|foo|bar|g'
sed 's@foo@bar@g'
sed 's:foo:bar:g'

The main reason to change it? Avoiding escaped slashes in paths and URLs.

# Hard to read
sed 's/\/var\/log\/nginx/\/data\/logs\/nginx/g' nginx.conf

# Much cleaner
sed 's#/var/log/nginx#/data/logs/nginx#g' nginx.conf

Choosing a delimiter: Pick a character that doesn’t appear in your pattern or replacement.

Delimiter	Good for	Watch out for
`#`	Paths	URL fragments, config comments
`\|`	Paths, URLs	No longer means regex alternation
`@`	URLs, paths	Emails, `user@host`
`:`	Paths	`https://`, ports, `PATH` values
`,`	General use	CSV

This becomes especially important when using variables:

# Breaks when $NEW_PATH contains '/'
sed "s/__DATA_DIR__/$NEW_PATH/" app.conf.tpl

# Better for paths
sed "s|__DATA_DIR__|$NEW_PATH|" app.conf.tpl

For arbitrary values, escape special characters first. Remember, & is special in the replacement regardless of the delimiter.

escaped=$(printf '%s' "$VAL" | sed 's/[&|\\]/\\&/g')
sed "s|__VALUE__|$escaped|" app.conf.tpl

A couple of gotchas:

An escaped delimiter is always literal. With . as the delimiter, \. means a literal dot, not “any character.” Avoid regex metacharacters (. * [ ^ $) as delimiters.
Multibyte characters aren’t allowed as delimiters.

Custom delimiters also work with addresses and y:

For addresses, prefix the delimiter with \.

sed '\#/var/log#d' file
sed 'y#abc#xyz#'

Addresses: Choosing which lines to process

Without an address, sed processes every line. Addresses let you target specific lines by number, pattern, or range.

sed '5d' file                     # Line 5
sed -n '$p' file                  # Last line, like tail -1
sed -n '/ERROR/p' app.log         # Matching lines, like grep
sed '/debug/Id' app.log           # Case-insensitive (GNU)
sed '10,20d' file                 # Lines 10-20
seq 10 | sed -n '0~2p'            # Every 2nd line, starting at 0 (GNU)
sed -n '/Traceback/,+5p' app.log  # Match plus 5 following lines (GNU)
sed '0,/localhost/s//db.internal/' app.conf  # First occurrence in the file (GNU)
sed '/^#/!d' file                 # Keep only comments

Notice the empty regex // in the last GNU-specific example. It reuses the last matched regex (localhost).

Commands: What can replace `s`?

Substitution is only one of the things sed can do. Here are its other commands, grouped by purpose.

Editing text

sed 's/http:/https:/g' urls.txt                                   # s: Substitute
echo "hello" | sed 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/'  # y: Transliterate
sed '/^$/d' file                                                   # d: Delete the line
sed '/^\[mysqld\]/a max_connections = 500' my.cnf                  # a: Append after
sed '1i #!/usr/bin/env bash' script.sh                             # i: Insert before
sed '/^#\?PermitRootLogin/c PermitRootLogin no' /etc/ssh/sshd_config  # c: Replace the line
sed '/^password=/z' app.env                                        # z: Empty pattern space (GNU)

D is different from d: it deletes up to the first newline in the pattern space, then restarts the cycle without reading new input.

Combined with N and P, it can remove consecutive duplicate lines, just like uniq:

printf 'a\na\nb\nb\nb\nc\n' | sed '$!N;/^\(.*\)\n\1$/!P;D'
# a
# b
# c

Output

sed -n '100,120p' big.log             # p: Print (usually with -n)
sed -n '/ERROR/=' app.log             # =: Print line numbers
sed -n '$=' file                      # Count lines, like wc -l
printf 'a\tb\r\n' | sed -n l            # l: Show special characters -> a\tb\r$
sed -n '/FATAL/{F;=;p}' app.log       # F: Print filename (GNU)
sed -n '/ERROR/w errors.log' app.log  # w file: Write matching lines to a file

Reading input

seq 6 | sed 'n;d'                     # n: Print current line, load next, then delete
sed 'N;s/\n/ /' file                  # N: Append next line to pattern space
sed '/# BEGIN extra/r extra.conf' nginx.conf  # r file: Insert another file's contents
sed 'R b.txt' a.txt                   # R file: Insert one line per match (GNU)

Hold space: sed’s second buffer

The pattern space holds the current line, while the hold space acts as a scratchpad that persists across lines.

These commands move data between them:

Cmd	Action
`h`	Pattern space → hold space
`H`	Append pattern space → hold space
`g`	Hold space → pattern space
`G`	Append hold space → pattern space
`x`	Swap both spaces

Here are a few things you can do with them:

sed -n '/ERROR/{x;p;x;p};h' app.log     # Print the line before each error too

printf 'web1\nweb2\nweb3\n' | sed -n 'H;${x;s/\n/,/g;s/^,//;p}'
# web1,web2,web3

sed -n '1!G;h;$p' file                  # Reverse a file, like tac

Flow control

sed also supports branching and loops, which come in handy for more complex operations.

sed '/^#/b; s/foo/bar/' file    # b: Skip comments, substitute elsewhere
echo 1234567 | sed ':a;s/^\([0-9]\+\)\([0-9]\{3\}\)/\1,\2/;ta'   # t: Loop while s succeeds -> 1,234,567
sed '10q' file                  # q: Print and quit, like head -10
sed '/FATAL/q 3' app.log; echo $?   # Quit with a custom exit code
sed '/^__END__/Q' script.sh     # Q: Quit without printing (GNU)
sed '/^server {/,/^}/{ /^\s*#/!s#listen 80;#listen 8080;#g }' nginx.conf   # { }: Group commands

Other

echo "uptime" | sed e           # e: Execute pattern space as a shell command (GNU)
sed 'v 4.2' file                # v: Require GNU sed >= 4.2

⚠️ Never use e with untrusted input. It executes shell commands and can introduce command injection.

Flags after the `s` command

The substitution command has several flags that control how replacements work:

echo aaaa | sed 's/a/b/g'          # g: All matches -> bbbb
echo aaaa | sed 's/a/b/2'          # N: Only the Nth match -> abaa
echo aaaa | sed 's/a/b/2g'         # Ng: Nth match onward (GNU) -> abbb
sed -n 's/ERROR/[E]/p' app.log     # p: Print only changed lines
sed -n 's#listen 80;#listen 8080;#w /dev/stdout' nginx.conf   # w file: Write changed lines
sed 's/error/ERROR/gI' app.log     # I: Case-insensitive (GNU)
sed -n 's#http://#https://#gIp' urls.txt   # Combine flags

Command-line options

You can also change how sed runs using command-line options:

sed -n '/ERROR/p' app.log                                  # -n: Disable automatic printing
sed -i.bak 's/^max_connections.*/max_connections = 500/' my.cnf   # -i: Edit in place with backup
sed -E 's/v([0-9]+)\.([0-9]+)/\2.\1/' <<< "v1.20"           # -E/-r: Extended regex -> 20.1
sed -e 's/foo/bar/' -e '/^$/d' file                         # -e: Multiple scripts
sed -f cleanup.sed file                                     # -f: Read script from a file
sed -s -n '$p' /var/log/app/*.log                           # -s: Treat files separately
tail -f app.log | sed -u 's/ERROR/\x1b[31m&\x1b[0m/'        # -u: Unbuffered output

Putting it all together

Let’s use what we’ve learned to change the listening port inside the server block of nginx.conf, skip commented lines, and keep a backup.

sed -i.bak '/^server {/,/^}/{ /^\s*#/!s#listen 80;#listen 8080;#g }' nginx.conf

Here’s how it works:

Piece	Concept
`-i.bak`	Edit in place with backup
`/^server {/,/^}/`	Range address
`{ ... }`	Command grouping
`/^\s*#/!`	Negated regex address
`s#...#...#`	Substitute with custom delimiter
`g`	Global flag

Portability: GNU vs. BSD (macOS) sed

Not every sed feature works on every system.

Anything marked (GNU) above won’t work with default BSD sed. On macOS, install GNU sed with brew install gnu-sed and use gsed.

In-place editing uses different syntax:

# GNU
sed -i 's/a/b/' file

# BSD
sed -i '' 's/a/b/' file

BSD’s a, i, and c commands require the text on a new line after a backslash.
\s, \+, \?, and \| in basic regex are GNU extensions. For portable scripts, use -E with +, ?, and |, and [[:space:]] instead of \s.

Conclusion

s/a/b/g and s#a#b#g are the same command. The delimiter is simply whatever character follows s.

And that’s just the beginning. sed gives you addresses, commands, flags, and a second buffer to work with.

Once you get comfortable with these, you can replace a lot of grep | awk | head | tail pipelines with a single sed expression.