sed and awk

sed rewrites text one line at a time. awk splits each line into fields and runs small programs over them. After this lesson the reader can replace text, cut line ranges, edit files in place, reorder matched text, total and average columns, count by key, and format reports from the command line.

The examples use GNU sed 4.9 and mawk 1.3.4. All awk programs here stick to POSIX awk, so they also run in gawk and in the awk that ships with macOS. Examples share one directory. Run them in order, because later examples reuse files that earlier ones create.

Example 1: replace text with s

The s command is the core of sed. Its form is s/pattern/replacement/. This example creates a small file and replaces the first cat on each line.

rm -rf sedawk-demo && mkdir -p sedawk-demo && cd sedawk-demo
printf 'cat sat on the mat\nthe cat ran past the other cat\nno animals here\n' > pets.txt
sed 's/cat/dog/' pets.txt
echo 'a/b/c' | sed 's|/|:|'
echo 'a/b/c' | sed 's/\//:/'

dog sat on the mat
the dog ran past the other cat
no animals here
a:b/c
a:b/c

sed reads pets.txt line by line, applies the script to each line, and prints the result to standard output. The file itself does not change. Line 2 has two cat words and only the first became dog. Line 3 has no match and passes through unchanged.

The character after s is the delimiter. It is / by convention, but any character works. When the pattern contains /, such as a path, pick another delimiter (| or #) or escape each slash with a backslash. The second and third commands do the same thing. The version with | is easier to read.

By default the pattern is a POSIX basic regular expression. Characters such as ., *, ^, and $ keep their regular-expression meaning. Add -E for extended regular expressions, where +, ?, |, (), and {} work without backslashes. Example 6 uses -E.

Example 2: the global flag and other flags

A flag after the last delimiter changes how s behaves. The g flag replaces every match on the line, not only the first.

echo 'one fish two fish red fish' | sed 's/fish/cow/g'
echo 'one fish two fish red fish' | sed 's/fish/cow/2'
echo 'one fish two fish red fish' | sed 's/fish/cow/2g'
echo 'Fish fish FISH' | sed 's/fish/cow/Ig'
sed -n 's/cat/dog/gp' pets.txt

one cow two cow red cow
one fish two cow red fish
one fish two cow red cow
cow cow cow
dog sat on the mat
the dog ran past the other dog

The first command changes all three words. The second changes only the second fish. The third changes the second and the third. The fourth uses I to match Fish, fish, and FISH as the same word.

The last command combines -n with the p flag. The option -n turns off the automatic printing of every line. The p flag then prints only the lines where s changed something, so the line no animals here is gone. This pattern prints only the changed lines, which is useful when previewing a large edit.

Example 3: select lines with addresses and ranges

An address placed before a command limits the lines the command runs on. An address is a line number, $ for the last line, a regular expression between slashes, or a range of two addresses.

seq 1 10 > nums.txt
sed -n '3p' nums.txt
sed -n '2,4p' nums.txt
sed -n '$p' nums.txt
sed -n '/^[5-6]$/p' nums.txt
sed -n '8,$p' nums.txt
sed '2,4s/^/> /' nums.txt | head -n 5

3
2
3
4
10
5
6
8
9
10
1
> 2
> 3
> 4
5

The command p prints a line. Without -n, every line prints once automatically and p would print selected lines a second time. With -n, only the selected lines appear.

The last command shows that an address works with s as well. The replacement s/^/> / inserts > at the start of lines 2 to 4 only. Lines 1 and 5 stay as they were. The head -n 5 limits the display to the first five lines.

A range of two regular expressions can match several times in one file. Each time the start expression matches, the range opens again and stays open until the end expression matches.

printf 'a\nSTART\nb\nc\nEND\nd\nSTART\ne\n' | sed -n '/START/,/END/p'

START
b
c
END
START
e

The first block, START to END, prints in full. The second START opens a range that never closes, so it prints to the end of input.

Example 4: delete lines with d

The command d deletes the selected lines. It takes the same addresses as p. A common use is stripping comments and blank lines from a configuration file.

cat > app.conf <<'CONF'
# database settings
db.host=localhost

db.port=5432
  # indented comment
db.name=shop

# cache settings
cache.ttl=300
CONF
sed '/^#/d' app.conf
echo ---
sed -e '/^[[:space:]]*#/d' -e '/^[[:space:]]*$/d' app.conf
echo ---
sed '2,3d' nums.txt | head -n 3
echo ---
sed '$d' nums.txt | tail -n 2

db.host=localhost

db.port=5432
  # indented comment
db.name=shop

cache.ttl=300
---
db.host=localhost
db.port=5432
db.name=shop
cache.ttl=300
---
1
4
5
---
8
9

The first command deletes lines that start with #. The indented comment survives because ^# requires the # at column 1. Blank lines also survive.

The second command passes two scripts with -e. sed runs them in order on each line. [[:space:]]* matches any number of spaces or tabs, so the indented comment and the whitespace-only lines are removed. The class [[:space:]] is POSIX and works on both GNU and BSD sed. The shorthand \s is a GNU extension.

The third command deletes lines 2 and 3 of the numbers file. The fourth deletes the last line. When a line is deleted, sed starts the next cycle immediately, so later commands in the script do not see that line.

Separate commands with ; or newlines inside one script, as in '/^#/d;/^$/d'. Some BSD versions require a newline instead of ; after certain commands such as a, i, c, and r. Using separate -e options avoids the issue.

Example 5: edit a file in place with -i

Option -i writes the result back to the file instead of standard output. Because the original is overwritten, keep a backup while learning.

cp app.conf work.conf
sed -i.bak 's/localhost/db.internal/' work.conf
grep '^db.host' work.conf work.conf.bak
diff work.conf.bak work.conf

work.conf:db.host=db.internal
work.conf.bak:db.host=localhost
2c2
< db.host=localhost
---
> db.host=db.internal

The suffix attached to -i names the backup file. Here work.conf.bak holds the original and work.conf holds the edited version. The grep with two file names prefixes each match with its file name. The diff output shows the single changed line: < marks the old file, > marks the new one, and 2c2 means line 2 changed.

GNU and BSD sed treat -i differently.

On GNU sed, the suffix must be attached directly to -i. On BSD sed, the suffix is a separate argument, and an empty string means no backup. The form -i.bak works on both and is the portable choice. Without a backup, a wrong script destroys the original, so test the script without -i first and compare with diff.

Example 6: capture groups and the & symbol

Parentheses in the pattern capture the text they match. The replacement refers to captures as \1, \2, and so on. This reorders or reuses parts of a match. The symbol & stands for the whole match.

echo 'John Smith' | sed -E 's/([A-Za-z]+) ([A-Za-z]+)/\2, \1/'
echo '2026-10-03' | sed -E 's#([0-9]{4})-([0-9]{2})-([0-9]{2})#\3/\2/\1#'
echo 'total: 42 items' | sed -E 's/[0-9]+/[&]/'
echo 'key=value' | sed -E 's/^([^=]+)=(.*)$/\1 is "\2"/'
echo 'cats and cats' | sed 's/\(c\)ats/\1at/g'

Smith, John
03/10/2026
total: [42] items
key is "value"
cat and cat

The first command captures two words and swaps them. Each ([A-Za-z]+) is one group, numbered from the left by the position of its opening parenthesis. The second command turns an ISO date into day/month/year. It uses # as the delimiter because the replacement contains /.

The third command wraps the first number in brackets. & inserts the text matched by [0-9]+, so no group is needed. To write a literal &, escape it as \&.

The fourth command splits a key=value line. [^=]+ matches one or more characters that are not =. (.*)$ captures the rest of the line.

The last command uses basic regular expressions, where groups are written \( and \). This is the form that works without -E. With -E, plain parentheses are groups and \( is a literal parenthesis. Mixing the two styles is a common source of errors.

Example 7: awk fields and the -F option

awk reads each line, splits it into fields, and runs the program. Fields are $1, $2, and so on. $0 is the whole line. NF is the number of fields. By default, fields are separated by runs of spaces and tabs.

cat > staff.txt <<'DATA'
alice engineering 85000
bob sales 62000
carol engineering 95000
dave support 48000
erin sales 71000
frank engineering 78000
DATA
awk '{print $1, $3}' staff.txt | head -n 3
awk '{print $NF}' staff.txt | head -n 2
awk '{print NF}' staff.txt | head -n 1
echo 'a   b	c' | awk '{print $2}'

alice 85000
bob 62000
carol 95000
85000
62000
3
b

The program is the text in single quotes. The braces hold an action. With no pattern in front, the action runs on every line. The comma in print $1, $3 prints the two values separated by a space (the output field separator). $NF is the last field, because NF holds the field count and $ takes the field with that number. The last command shows that runs of spaces and tabs count as one separator.

Option -F sets a different field separator. The next example uses a colon-separated file in the style of /etc/passwd and a comma-separated one.

cat > users.txt <<'DATA'
root:x:0:0:root:/root:/bin/bash
alice:x:1000:1000:Alice:/home/alice:/bin/zsh
bob:x:1001:1001:Bob:/home/bob:/bin/bash
DATA
awk -F: '{print $1, $7}' users.txt
printf 'id,name,city\n1,Ana,Lyon\n2,Ben,Oslo\n' > people.csv
awk -F, -v OFS=' | ' '{print $1, $3}' people.csv
awk -F'[:/]' '{print $1, $NF}' users.txt
echo 'a b c' | awk '{$2 = "X"; print}'
echo 'a   b   c' | awk '{$1 = $1; print}'

root /bin/bash
alice /bin/zsh
bob /bin/bash
id | city
1 | Lyon
2 | Oslo
root bash
alice zsh
bob bash
a X c
a b c

-F: sets the separator to a colon. -F, sets it to a comma. The option -v OFS=' | ' sets the output field separator, so the printed fields join with |. The separator in -F is a regular expression when it has more than one character, so -F'[:/]' splits on a colon or a slash. The simple -F, handles only plain CSV. Quoted fields that contain commas need a real CSV parser.

Assigning to a field rebuilds $0 with the output field separator. $2 = "X" replaces the second field. The assignment $1 = $1 changes nothing visible, but it forces awk to rebuild the line, which collapses repeated spaces into single spaces.

Example 8: patterns, BEGIN, and END

An awk program is a list of pattern { action } pairs. The action runs only on lines where the pattern is true. A pattern with no action prints the matching line. BEGIN runs once before any input. END runs once after all input.

awk '$3 > 70000' staff.txt
awk '$2 == "sales" {print $1}' staff.txt
awk '/eng/ {print $1}' staff.txt
awk '$2 ~ /^s/ {print $1, $2}' staff.txt
awk 'NR == 2' staff.txt
awk 'NR >= 2 && NR <= 3' staff.txt
awk 'BEGIN {print "name salary"} {print $1, $3} END {print "rows:", NR}' staff.txt

alice engineering 85000
carol engineering 95000
erin sales 71000
frank engineering 78000
bob
erin
alice
carol
frank
bob sales
dave support
erin sales
bob sales 62000
bob sales 62000
carol engineering 95000
name salary
alice 85000
bob 62000
carol 95000
dave 48000
erin 71000
frank 78000
rows: 6

NR is the number of the current input line, counted across all input files. FNR restarts at 1 for each file. Awk compares two values as numbers when both look like numbers, and as strings otherwise. That is why $3 > 70000 works without conversion.

The last command shows the three parts together. BEGIN prints a header before line 1. The middle action runs for every line. END prints the line count, using NR, which still holds the last value after input ends.

Example 9: sums and averages

Awk variables need no declaration. An unset variable is 0 in a number context and empty in a string context. A running total is one statement: sum += $3. The END block prints it.

awk '{sum += $3} END {print "total:", sum}' staff.txt
awk '{sum += $3} END {print "average:", sum / NR}' staff.txt
awk 'NR == 1 || $3 > max {max = $3; who = $1} END {print who, max}' staff.txt
awk '$2 == "engineering" {sum += $3; n++} END {if (n > 0) print sum / n; else print "none"}' staff.txt
printf 'id,qty\n1,4\n2,6\n3,5\n' | awk -F, 'NR > 1 {sum += $2} END {print sum}'

total: 439000
average: 73166.7
carol 95000
86000
15

The first command adds field 3 of every line. The second divides the total by NR, the line count at the end. The third tracks the largest value: on line 1, or on any line where field 3 exceeds the current maximum, it stores the value and the name. Starting with NR == 1 avoids relying on an initial value.

The fourth command averages only engineers. The counter n counts the matching lines. The if guards the division, because dividing by zero is an error in awk. The last command skips a header with NR > 1.

Numbers print with up to six significant digits by default (OFMT). To control the decimals, use printf, covered in Example 11. Values that are integers print as integers.

Example 10: count and group by key with arrays

Awk arrays are associative. The index can be any string. Reading a missing element creates it with the value 0 (or empty). That makes count[$3]++ a one-line counter. The following examples use a small web log.

cat > web.log <<'LOG'
2026-10-03 10:00:01 INFO web GET /home 200 12
2026-10-03 10:00:02 INFO web GET /api/users 200 48
2026-10-03 10:00:05 WARN web GET /api/orders 404 9
2026-10-03 10:00:07 ERROR db POST /api/orders 500 230
2026-10-03 10:00:09 INFO web GET /home 200 11
2026-10-03 10:00:12 INFO web GET /api/users 200 51
2026-10-03 10:00:15 ERROR web POST /api/login 500 310
2026-10-03 10:00:18 INFO web GET /home 304 4
2026-10-03 10:00:21 WARN db GET /api/orders 200 120
2026-10-03 10:00:25 INFO web GET /api/users 200 44
LOG
awk '{count[$3]++} END {for (k in count) print k, count[k]}' web.log | sort
awk '{sum[$6] += $8; n[$6]++} END {for (p in sum) print p, n[p], sum[p] / n[p]}' web.log | sort
awk '!seen[$6]++ {print $6}' web.log

ERROR 2
INFO 6
WARN 2
/api/login 1 310
/api/orders 3 119.667
/api/users 3 47.6667
/home 3 9
/home
/api/users
/api/orders
/api/login

Fields in the log are date, time, level, service, method, path, status, and milliseconds, so $3 is the level, $6 the path, and $8 the duration.

The first command counts lines per level. count[$3]++ adds one to the element named by the level. The END block loops with for (k in count), which visits each key once.

The order of for (k in array) is not defined. It differs between awk implementations. Pipe the output to sort to get a stable order, as these commands do.

The second command keeps two arrays with the same key: the total milliseconds and the request count. Dividing them gives the average per path. The third command prints each path once, in first-seen order. The expression !seen[$6]++ is true the first time a path appears: the old value is 0, !0 is true, and the increment makes later values nonzero. It removes duplicates without sorting, which sort -u cannot do.

Keys can combine fields. count[$5 " " $7]++ counts by method and status together.

Example 11: format output with printf

The print statement joins values with spaces. The printf statement takes a format string and does not add a newline, so end the format with \n. It controls width, alignment, and decimals.

awk '{printf "%-10s %-12s %8d\n", $1, $2, $3}' staff.txt | head -n 3
awk '{sum[$2] += $3; n[$2]++} END {for (d in sum) printf "%-12s %2d %10.2f\n", d, n[d], sum[d] / n[d]}' staff.txt | sort
awk 'END {printf "errors: %d of %d (%.1f%%)\n", e, NR, 100 * e / NR} $3 == "ERROR" {e++}' web.log
awk 'BEGIN {printf "%05d|%x|%c|%s\n", 42, 255, 65, "end"}'

alice      engineering     85000
bob        sales           62000
carol      engineering     95000
engineering   3   86000.00
sales         2   66500.00
support       1   48000.00
errors: 2 of 10 (20.0%)
00042|ff|A|end

The first command prints a table with aligned columns. The - flag left-aligns the text, and numbers right-align by default. The second command builds a report by department, with a count and an average to two decimals. The third prints a percentage. %% prints one %. The program has the END block first and the pattern rule second, which works because END always runs last. The fourth command shows zero padding, hexadecimal, and a character from the code 65.

Each conversion in the format consumes one argument, in order, so the number of arguments must match the number of conversions.

Example 12: choose among grep, sed, and awk

The three tools overlap. Each task has a natural fit. This example counts the ERROR lines in the log with each tool.

grep -c ERROR web.log
sed -n '/ERROR/p' web.log | wc -l
awk '/ERROR/ {n++} END {print n}' web.log
grep ERROR web.log | sed -E 's/^[^ ]+ //' | awk '{print $1, $4, $6}'

2
2
2
10:00:07 POST 500
10:00:15 POST 500

All three first commands report 2. The fourth is a pipeline that uses each tool for what it does best. grep selects the lines. sed strips the date, which is the first word and its trailing space: ^[^ ]+ matches non-space characters from the start of the line, followed by one space. awk then picks fields. After the date is removed, the fields shift left by one, so $1 is the time, $4 is the method, and $6 is the status.

The criteria are the unit of work. When the question is "which lines", use grep. When the question is "change this text", use sed. When the question is about values in columns, use awk. Each tool also handles a smaller task of the next one, so a pipeline of three tools is not required when one awk program covers the job.

Other details matter in practice. sed has no numeric comparison or variables beyond its small hold space, so conditions on values need awk. awk programs that grow past a few lines belong in a file run with awk -f script.awk.

Interview points

• sed 's/a/b/' replaces the first match on each line. The g flag replaces all matches, and a number replaces only the Nth.
• sed prints every line by default. -n suppresses that, and the p command or p flag prints selected lines.
• sed -i edits in place. GNU needs -i or -i.bak. BSD needs a separate suffix argument, -i '' for none. -i.bak is portable.
• Basic regular expressions need \(, \), and \{ for groups and intervals. -E switches to extended syntax where bare parentheses group.
• Captures are \1 to \9. & is the entire match.
• In awk, $0 is the line, $1 to $NF are fields, and NF and NR are the field and line counters. FNR restarts per file.
• -F sets the input field separator, -v OFS= sets the output separator, and assigning $1 = $1 rebuilds the line.
• BEGIN runs before input and END after it. Totals and averages are printed in END. Guard divisions with an if.
• Awk arrays are associative and for (k in a) has no defined order, so sort the output.
• !seen[$1]++ prints the first occurrence of each key without sorting.
• Choose grep for selecting lines, sed for editing text, and awk for column logic and arithmetic.
• gensub, IGNORECASE, and asort are gawk extensions. Portable scripts avoid them.

Summary

• sed is a stream editor: it reads a line, applies the script, and prints the result. The input file is not changed without -i.
• s/pattern/replacement/flags is the substitution command. Any character can be the delimiter.
• Flags: g for all matches, a number for the Nth match, I for case-insensitive (GNU), p to print changed lines.
• Addresses are a line number, $, /regex/, or a range a,b. They apply to any command, including s.
• d deletes lines and p prints them. Use -n with p so lines print once.
• -i.bak edits in place and keeps a backup on both GNU and BSD sed.
• -E enables extended regular expressions. In basic mode, groups are \( \).
• \1 to \9 refer to captured groups in the replacement, and & is the whole match.
• awk splits each line into fields $1, $2, and so on. $NF is the last field and $0 is the whole line.
• -F sets the input separator. -v name=value sets a variable before the program runs.
• An awk program is pattern { action } pairs. BEGIN and END run once, before and after the input.
• Variables start at 0 or empty. sum += $3 totals a column, and sum / NR gives an average.
• Arrays are associative: count[$3]++ counts by key, and for (k in count) visits keys in an undefined order.
• printf takes a format with %s, %d, %.2f, width, and alignment. It does not add a newline.
• grep selects lines, sed edits text, and awk computes over fields.