xargs and jq

Pipes pass text from one command to the next, but many commands read arguments, not standard input. xargs closes that gap by turning input lines into arguments. jq does the same job for JSON: it reads structured data, selects parts of it, and prints the result as text that the rest of a pipeline can use.

All output in this lesson comes from GNU xargs 4.9.0 and jq 1.7 on Linux. Notes on macOS appear where behavior differs. The last section of examples starts a local web server on 127.0.0.1 so that the curl example works without internet access.

Example 1: xargs basics

The smallest use of xargs reads whitespace-separated words from standard input and passes them as arguments to a command. With no command given, xargs runs echo.

printf 'alpha\nbeta\ngamma\n' | xargs echo
printf 'alpha\nbeta\ngamma\n' | xargs echo prefix:

alpha beta gamma
prefix: alpha beta gamma

The first command receives three lines and runs echo alpha beta gamma once. The line breaks in the input are treated as spaces, so the output is one line. In the second command, the words after xargs (echo prefix:) form the fixed start of the command. xargs appends the input words after them.

This is the difference between a pipe and xargs. A pipe feeds standard input. xargs feeds the command line. Commands such as rm, mkdir, cp, mv, git add, and kill take their targets as arguments, so they need xargs to consume a list from a pipeline.

By default xargs splits input on spaces, tabs, and newlines, and it also interprets quotes and backslashes in the input. Both rules cause trouble with file names. Example 4 shows the safe alternative.

GNU xargs runs the command once with no arguments when the input is empty. The option -r (--no-run-if-empty) prevents that. BSD xargs on macOS has its own rules for empty input, so check man xargs there before relying on either behavior.

Example 2: -n limits arguments per run

A command line has a size limit, and some commands should receive a fixed number of arguments per run. The option -n N caps the number of arguments, and xargs starts as many runs as it needs.

printf '1 2 3 4 5 6 7\n' | xargs -n 3 echo

1 2 3
4 5 6
7

Seven words with -n 3 produce three runs: echo 1 2 3, echo 4 5 6, and echo 7. The last run receives the remainder.

Use -n 1 when a command accepts only one argument, for example a script that takes one host name. Without -n, xargs packs as many arguments as the system limit allows into each run. That limit is why xargs can process a million file names where a shell glob such as rm *.log fails with "Argument list too long".

Example 3: -I {} places the input

By default xargs appends arguments at the end of the command. The option -I {} names a placeholder, and xargs replaces every {} in the command with one input line. This allows an argument in the middle of a command or repeated several times.

printf 'a.txt\nb.txt\n' | xargs -I {} sh -c 'echo "copy {} to {}.bak"'

copy a.txt to a.txt.bak
copy b.txt to b.txt.bak

The placeholder appears twice in the command, and both are replaced. The text {} is only a convention. Any string works, for example -I @, which helps when the command itself contains braces.

-I changes two behaviors. It runs the command once per input line, as if -n 1 were set. It also splits input on newlines only, not on spaces, so a line that contains spaces stays whole. The same option makes a real copy command:

mkdir -p backup
touch a.txt b.txt
printf 'a.txt\nb.txt\n' | xargs -I {} cp {} backup/{}.bak
ls backup

a.txt.bak
b.txt.bak

Each run executed cp a.txt backup/a.txt.bak and cp b.txt backup/b.txt.bak. On macOS, xargs -I works the same way. BSD xargs also offers -J for a placeholder that keeps multiple arguments together, which GNU xargs does not have.

Example 4: -0 with find -print0

File names can contain spaces and newlines. A default xargs splits my file.txt into two arguments. The fix is to separate names with the NUL byte, which cannot appear in a file name. find -print0 writes NUL-separated names, and xargs -0 reads them.

touch "my file.txt" "plain.txt"
find . -maxdepth 1 -name '*.txt' | xargs ls -l 2>&1 | head -3
echo ---
find . -maxdepth 1 -name '*.txt' -print0 | xargs -0 ls -1
echo ---
find . -maxdepth 1 -name '*.txt' -print0 | xargs -0 -n 1 echo item:

ls: cannot access './my': No such file or directory
ls: cannot access 'file.txt': No such file or directory
-rw-r--r-- 1 root root 0 Oct  3 15:18 ./a.txt
---
./a.txt
./b.txt
./my file.txt
./plain.txt
---
item: ./b.txt
item: ./my file.txt
item: ./plain.txt
item: ./a.txt

The first pipeline breaks the name into ./my and file.txt, and ls reports two missing files. The second pipeline uses -print0 and -0, and ls receives ./my file.txt as one argument. The third pipeline adds -n 1 to show one run per name.

Two details depend on the machine. The order of names from find follows the directory order on disk, so the third block can be ordered differently on another system. The timestamp and owner in the ls -l line also vary.

Rule: when file names come from find, use -print0 and -0 together. GNU find can also run commands itself with -exec cmd {} +, which is equivalent and needs no xargs. The pipeline form is useful when the command after xargs is grep, sort, or another tool in a longer pipeline. BSD xargs and BSD find on macOS both support -0 and -print0.

Example 5: -P runs commands in parallel

The option -P N runs up to N commands at the same time. It helps when each run waits on something slow, such as a network request or a compression job.

printf '1\n2\n3\n4\n' | xargs -P 4 -I {} sh -c 'sleep 1; echo done {}' > par.txt
echo "parallel lines: $(wc -l < par.txt)"
sort par.txt

parallel lines: 4
done 1
done 2
done 3
done 4

Four jobs that each sleep one second finish in about one second with -P 4. The order in which the jobs print is not fixed, because whichever finishes first writes first. The example therefore sorts the lines before showing them. Any script that depends on output order needs the same treatment, or each job should write to its own file.

A timing comparison confirms the effect. The script measured whole seconds with date +%s:

s=$(date +%s); printf '1\n2\n3\n4\n' | xargs -P 1 -I {} sleep 1; e=$(date +%s)
echo "P1 elapsed at least 4s: $([ $((e-s)) -ge 4 ] && echo yes)"
s=$(date +%s); printf '1\n2\n3\n4\n' | xargs -P 4 -I {} sleep 1; e=$(date +%s)
echo "P4 elapsed at most 2s: $([ $((e-s)) -le 2 ] && echo yes)"

P1 elapsed at least 4s: yes
P4 elapsed at most 2s: yes

-P 1 is the default and runs jobs one after another, so four one-second jobs take four seconds. With -P 4 they overlap. The exact elapsed times are not printed because they vary between runs.

Choose a value close to the number of CPU cores for CPU-bound work. Choose a higher value for work that waits on a network. Jobs that write to the same file at the same time can interleave their output, so give each job its own output. nproc prints the core count on Linux, and sysctl -n hw.ncpu does on macOS.

Example 6: jq . pretty-prints JSON

jq takes a filter and applies it to JSON input. The simplest filter is ., which means "the whole input". It prints the data with indentation and keeps the key order of the input.

echo '{"a":1,"b":[true,null],"c":"x"}' | jq .
echo '{"a":1,"b":[true,null],"c":"x"}' | jq -c .

{
  "a": 1,
  "b": [
    true,
    null
  ],
  "c": "x"
}
{"a":1,"b":[true,null],"c":"x"}

The first command expands the document across lines. The second adds -c (compact), which prints each result on one line. Compact output is useful when one JSON value per line feeds grep, sort, or wc -l.

When the input is not valid JSON, jq prints an error and exits with status 5:

echo '{bad' | jq .
echo "exit=$?"

jq: parse error: Invalid numeric literal at line 2, column 0
exit=5

jq is not part of a default Linux or macOS install. Install it with apt install jq on Debian and Ubuntu or brew install jq on macOS. The remaining jq examples use this file, created once:

cat > orders.json <<'J'
{
  "store": "north",
  "open": true,
  "orders": [
    {"id": 1, "customer": "ana", "total": 25.5, "status": "paid", "tags": ["gift", "web"]},
    {"id": 2, "customer": "ben", "total": 80, "status": "pending", "tags": ["web"]},
    {"id": 3, "customer": "ana", "total": 12.25, "status": "paid", "tags": []},
    {"id": 4, "customer": "cy", "total": 140, "status": "refunded", "tags": ["phone", "web"]},
    {"id": 5, "customer": "ben", "total": 60, "status": "paid", "tags": ["phone"]}
  ]
}
J

Example 7: field access

A filter written as .name reads one field of an object. Fields chain with more dots to reach nested data.

jq .store orders.json
jq '.orders[0].customer' orders.json
jq '.missing' orders.json
jq '.orders[0] | .tags[0]' orders.json

"north"
"ana"
null
"gift"

.store returns the string "north", and the quotes remain because the output is valid JSON. .orders[0].customer goes into the orders field, takes the first item, and reads its customer. A field that does not exist produces null, not an error, so a misspelled name yields null silently. The last command uses the pipe operator | inside the filter. It sends the result of the left side into the right side, the same way a shell pipe does, but within one jq program.

Quote the filter in single quotes. Characters such as [, ], |, and $ have meaning to the shell, and single quotes pass them through unchanged. A filter that is only .store has no special characters, so the quotes are optional there.

Example 8: arrays

Square brackets index an array. Index 0 is the first item, a negative index counts from the end, and [a:b] slices. The form [] with nothing inside produces every item.

jq '.orders[1]' orders.json
jq '.orders[-1].id' orders.json
jq '.orders[1:3] | .[].id' orders.json
jq -c '.orders[].tags' orders.json

{
  "id": 2,
  "customer": "ben",
  "total": 80,
  "status": "pending",
  "tags": [
    "web"
  ]
}
5
2
3
["gift","web"]
["web"]
[]
["phone","web"]
["phone"]

.orders[1] returns the second order as an object. .orders[-1].id returns the id of the last order, which is 5. .orders[1:3] returns items at positions 1 and 2 (the end index is excluded), and .[].id then prints the id of each, giving 2 and 3.

The last command shows how [] behaves. .orders[].tags does not return one array. It emits one result per order, and jq prints each result separately. The filter runs once for every item, and the output is a stream of values. This stream model is the central idea in jq: most filters take one value and emit zero or more values, and a pipe feeds each emitted value to the next stage.

Example 9: select filters items

select(condition) keeps a value when the condition is true and drops it otherwise. Combined with .[], it filters an array.

jq -c '.orders[] | select(.status == "paid")' orders.json | head -2
jq '.orders[] | select(.total > 50 and .status != "refunded") | .id' orders.json

{"id":1,"customer":"ana","total":25.5,"status":"paid","tags":["gift","web"]}
{"id":3,"customer":"ana","total":12.25,"status":"paid","tags":[]}
2
5

The first command emits each order and keeps the ones whose status equals "paid". head -2 shows the first two of the three matches. The second command combines two tests with and, keeps orders over 50 that are not refunded, and prints only their ids. Order 4 has a total of 140 but a status of refunded, so it is excluded.

select is the jq counterpart of grep for structured data. It compares fields, not text lines, so "paid" inside a different field never causes a false match.

Example 10: map transforms an array

map(f) applies a filter f to every item of an array and collects the results into a new array. It is shorthand for [.[] | f].

jq -c '.orders | map(.total)' orders.json
jq '.orders | map(.total) | add' orders.json
jq -c '.orders | map(select(.status=="paid")) | map(.id)' orders.json

[25.5,80,12.25,140,60]
317.75
[1,3,5]

The first command extracts the total of each order into an array. The second sends that array to add, which sums numbers, giving 317.75. The third shows that map accepts any filter, including select. It keeps only paid orders and then maps them to their ids.

The difference from .[] is the output shape. .orders[] | .total emits five separate numbers. .orders | map(.total) returns one array. Choose map when the next step needs the whole array, such as add, length, sort, min, or max.

Example 11: -r prints raw text

By default jq prints strings as JSON, with quotes. The option -r (--raw-output) prints string results without quotes, which is what shell tools need.

jq '.orders[].customer' orders.json | head -2
jq -r '.orders[].customer' orders.json | head -2
jq -r '.orders[] | [.id, .customer, .total] | @tsv' orders.json
jq -r '.orders[] | [.id, .customer] | @csv' orders.json | head -2

"ana"
"ben"
ana
ben
1	ana	25.5
2	ben	80
3	ana	12.25
4	cy	140
5	ben	60
1,"ana"
2,"ben"

Without -r the names carry quotes. With -r they are plain words, ready for a loop, sort, or xargs. The format operators @tsv and @csv turn an array into one tab-separated or comma-separated line, so jq can produce a table that column -t, awk, or a spreadsheet reads. @csv quotes strings and leaves numbers bare. Other operators include @json, @uri, @sh (shell-quoted), and @base64.

String interpolation builds readable lines directly. The text "\(.customer) paid \(.total)" places field values inside a string. Example 15 uses it.

Example 12: building objects

A filter that looks like an object literal builds a new object. This reshapes data: keep some fields, rename others, and compute new ones.

jq -c '.orders[] | {id, who: .customer, big: (.total > 50)}' orders.json
jq -c '{store, count: (.orders | length)}' orders.json

{"id":1,"who":"ana","big":false}
{"id":2,"who":"ben","big":true}
{"id":3,"who":"ana","big":false}
{"id":4,"who":"cy","big":true}
{"id":5,"who":"ben","big":true}
{"store":"north","count":5}

In the first command, {id, ...} is shorthand for {id: .id}. The entry who: .customer renames a field. The entry big: (.total > 50) computes a boolean. Wrap a computed value in parentheses so jq reads the whole expression as the value. The second command builds one object from the top-level data: store is copied and count holds the length of the orders array.

Wrap the loop in brackets to collect results into one array instead of a stream: [.orders[] | {id, who: .customer}]. This is the same result as .orders | map({id, who: .customer}).

Example 13: length

length returns a size that depends on the type of its input.

jq '.orders | length' orders.json
jq '.store | length' orders.json
jq -c '.orders[] | {id, n: (.tags | length)}' orders.json | head -3
jq '.missing | length' orders.json

5
5
{"id":1,"n":2}
{"id":2,"n":1}
{"id":3,"n":0}
0

The first result counts the five orders. The second counts the five characters in "north", which has the same value by coincidence and shows that length depends on the type. The third command counts the tags of each order, and order 3 has an empty array, so its count is 0. The last command applies length to the null that .missing returns, and the result is 0.

Because a missing field becomes null and null | length is 0, a count of 0 can mean either an empty array or a field that does not exist. To tell the two apart, test the field first, for example with has("orders") on the parent object.

Example 14: group_by

group_by(f) sorts an array by the value of f and splits it into sub-arrays, one per distinct value. It works like sort | uniq -c for structured data, with access to every field of the grouped items.

jq -c '.orders | group_by(.customer) | map({customer: .[0].customer, orders: length, spent: (map(.total) | add)})' orders.json
jq -c '.orders | group_by(.status) | map({(.[0].status): length}) | add' orders.json

[{"customer":"ana","orders":2,"spent":37.75},{"customer":"ben","orders":2,"spent":140},{"customer":"cy","orders":1,"spent":140}]
{"paid":3,"pending":1,"refunded":1}

In the first command, group_by(.customer) returns an array of three groups: ana's two orders, ben's two orders, and cy's one order. The following map runs once per group. Inside it, .[0].customer reads the customer from the first item of the group, length counts the items, and map(.total) | add sums them. Ana's total is 25.5 plus 12.25, which is 37.75.

The second command builds a count per status. For each group it makes a one-key object. The parentheses in {(.[0].status): length} tell jq to use the value of the expression as the key name. add then merges the five small objects into one. The groups come out sorted by the grouping value, so the output order is stable.

Example 15: piping curl JSON into jq

Most web APIs return JSON, and curl -s writes the response body to standard output, so the two commands combine in a pipeline. The option -s (silent) hides the progress meter, which would otherwise mix with the data on a terminal.

This example does not need internet access. It serves the file from Example 6 with Python's built-in web server, bound to 127.0.0.1 so that only the local machine can connect, and stops the server afterwards.

mkdir -p www && cp orders.json www/
(cd www && python3 -m http.server 8765 --bind 127.0.0.1 >/dev/null 2>&1 & echo $! > ../srv.pid)
sleep 1
curl -s http://127.0.0.1:8765/orders.json | jq -r '.orders[] | select(.status=="paid") | "\(.customer) paid \(.total)"'
curl -s http://127.0.0.1:8765/orders.json | jq '.orders | map(.total) | add'
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8765/orders.json
curl -s http://127.0.0.1:8765/orders.json | jq -r '.orders[].customer' | sort -u | xargs -I {} echo "customer: {}"
kill $(cat ../srv.pid)

ana paid 25.5
ana paid 12.25
ben paid 60
317.75
200
customer: ana
customer: ben
customer: cy

The first pipeline downloads the JSON, keeps paid orders, and prints one line per order with -r so the text has no quotes. The second sums all totals. The third command skips the body with -o /dev/null and prints only the HTTP status code with -w '%{http_code}\n', which is a way to check that an endpoint answers before parsing it. The fourth command joins both tools from this lesson: jq -r extracts the names, sort -u removes duplicates, and xargs -I {} runs one command per unique name. The last line stops the server by process id.

Calls to real APIs follow the same shape, with extra headers such as curl -s -H "Authorization: Bearer $TOKEN" https://api.example.com/items | jq .... Output from internet services varies and is not shown here. If a request fails, curl -s prints nothing, and jq then reports nothing or a parse error. Add -f (--fail) so curl returns a non-zero exit code on HTTP errors, and check ${PIPESTATUS[0]} in Bash to read it from inside a pipeline.

Interview points

• A pipe sends data to standard input. xargs converts input lines into command-line arguments, which is why rm, cp, and kill need it.
• Default xargs splits on whitespace and interprets quotes. Use find -print0 | xargs -0 for file names, because NUL cannot appear in a path.
• -n N sets arguments per run. -I {} runs once per line and allows a placeholder in any position, and it splits on newlines only.
• xargs exists partly because of the argument-length limit. A glob that expands past the limit fails, while xargs splits the list across several runs.
• -P N runs jobs in parallel. Output order is not deterministic, and jobs that write to one file can interleave their writes.
• GNU xargs runs the command once on empty input unless -r is set. BSD xargs differs, so portable scripts check the platform or guard the input.
• jq filters take one value and emit zero or more. .[] emits each array item, and a pipe inside a filter feeds each result onward.
• A missing field yields null, not an error, so typos fail silently and null | length is 0.
• -r prints strings without quotes, -c prints compact one-line JSON, and @tsv and @csv produce table text.
• map(f) equals [.[] | f]. select(cond) filters. group_by(f) returns sub-arrays sorted by the key.
• curl -s | jq is the standard way to inspect an API response. Use curl -f to fail on HTTP errors, and -w '%{http_code}' to read the status.

Summary

• xargs builds command lines from standard input and runs them, in as many batches as the system limit requires.
• With no command, xargs runs echo. Arguments from input are appended after the fixed command words.
• -n N limits arguments per run, and -I {} substitutes each input line into a placeholder, one run per line.
• -0 reads NUL-separated input, and find -print0 | xargs -0 handles names with spaces and newlines.
• -P N runs up to N jobs at once. Sort or separate the output when order matters.
• jq . pretty-prints JSON, -c compacts it, and invalid input exits with status 5.
• .field, .a.b, and .arr[0] read data. A missing field gives null.
• .[] streams array items, [a:b] slices, and negative indexes count from the end.
• select(cond) filters a stream and map(f) transforms every item of an array into a new array.
• -r removes quotes from strings, and @tsv, @csv, and "\(.x)" interpolation produce shell-ready text.
• {id, who: .customer} builds objects, length counts items, keys, or characters, and group_by(f) splits an array by key.
• curl -s URL | jq FILTER parses an HTTP JSON response, and a local python3 -m http.server provides a safe target for practice.