LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

jq: The Command-Line JSON Processor You Should Already Know

jqjsonclilinuxdevopsscriptingtools

If you work with APIs, cloud CLIs, or any modern infrastructure tooling, you deal with JSON constantly. jq is to JSON what awk is to text — a purpose-built language for slicing, filtering, and transforming structured data at the command line. This guide goes from first principles to the patterns you’ll actually reach for in production.

Installation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Debian / Ubuntu
apt-get install -y jq

# RHEL / Rocky / Fedora
dnf install -y jq

# macOS
brew install jq

# Alpine
apk add jq

# Check version
jq --version
# jq-1.7.1

The Mental Model

jq is a filter language. Every jq program is a filter that takes input JSON, transforms it, and produces output JSON. You chain filters together with the pipe | operator, exactly like a Unix pipeline.

1
2
3
4
5
6
7
8
9
# Basic invocation
echo '{"name":"alice","age":30}' | jq '.name'
# "alice"

# From a file
jq '.name' data.json

# From a URL (with curl)
curl -s https://api.example.com/users/1 | jq '.'

jq outputs valid, pretty-printed JSON by default. Use -r (raw) to strip quotes from strings for use in shell scripts.


Basic Filters

Identity — .

The simplest filter. Returns its input unchanged, but pretty-prints it:

1
2
3
4
5
echo '{"a":1,"b":2}' | jq '.'
# {
#   "a": 1,
#   "b": 2
# }

Field Access — .field

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
echo '{"user":{"name":"alice","role":"admin"}}' | jq '.user'
# {
#   "name": "alice",
#   "role": "admin"
# }

jq '.user.name'
# "alice"

# Optional field (no error if field missing)
jq '.user.email?'
# null

Array Index — .[N]

1
2
3
4
5
6
7
8
echo '[10,20,30,40]' | jq '.[0]'
# 10

jq '.[-1]'   # Last element
# 40

jq '.[1:3]'  # Slice (index 1 up to but not including 3)
# [20, 30]

Array / Object Iterator — .[]

Explodes an array or object into a stream of values:

1
2
3
4
5
6
7
8
echo '[1,2,3]' | jq '.[]'
# 1
# 2
# 3

echo '{"a":1,"b":2}' | jq '.[]'
# 1
# 2

Pipes and Chaining

Pipe | passes the output of one filter as input to the next:

1
2
3
4
echo '{"users":[{"name":"alice"},{"name":"bob"}]}' \
  | jq '.users[] | .name'
# "alice"
# "bob"

This is the core pattern: navigate to a collection, explode it, then extract a field.


Constructing Output

Arrays — [expr]

Wrap an expression in [] to collect its output into an array:

1
2
3
4
5
6
echo '{"users":[{"name":"alice","age":30},{"name":"bob","age":25}]}' \
  | jq '[.users[] | .name]'
# [
#   "alice",
#   "bob"
# ]

Objects — {key: expr}

Build new objects from pieces of the input:

1
2
3
4
5
6
echo '{"name":"alice","age":30,"email":"alice@example.com","role":"admin"}' \
  | jq '{username: .name, isAdmin: (.role == "admin")}'
# {
#   "username": "alice",
#   "isAdmin": true
# }

When the key name matches the field name, use shorthand:

1
2
jq '{name, age}'
# { "name": "alice", "age": 30 }

Combining Both

1
2
# Reshape an array of objects
jq '[.users[] | {name, admin: (.role == "admin")}]'

Useful Built-in Functions

length

1
2
3
4
5
6
7
8
echo '[1,2,3,4,5]' | jq 'length'
# 5

echo '"hello"' | jq 'length'
# 5

echo '{"a":1,"b":2}' | jq 'length'
# 2

keys and values

1
2
3
4
5
6
7
8
echo '{"b":2,"a":1,"c":3}' | jq 'keys'
# ["a","b","c"]    (sorted alphabetically)

jq 'values'
# [1,2,3]

jq 'keys_unsorted'
# ["b","a","c"]   (original order)

has and in

1
2
3
4
5
echo '{"name":"alice"}' | jq 'has("name")'
# true

jq 'has("email")'
# false

type

1
2
3
4
5
6
echo '"hello"' | jq 'type'  # "string"
echo '42'      | jq 'type'  # "number"
echo 'null'    | jq 'type'  # "null"
echo '[]'      | jq 'type'  # "array"
echo '{}'      | jq 'type'  # "object"
echo 'true'    | jq 'type'  # "boolean"

select — Conditional Filtering

select(expr) passes through values where expr is true, drops the rest:

1
2
3
4
5
6
7
8
9
echo '[1,2,3,4,5,6]' | jq '.[] | select(. > 3)'
# 4
# 5
# 6

# Filter objects in an array
echo '[{"name":"alice","age":30},{"name":"bob","age":17}]' \
  | jq '[.[] | select(.age >= 18)]'
# [{"name":"alice","age":30}]

map and map_values

map(f) is shorthand for [.[] | f]:

1
2
3
4
5
6
7
8
9
echo '[1,2,3,4]' | jq 'map(. * 2)'
# [2,4,6,8]

echo '[{"name":"alice"},{"name":"bob"}]' | jq 'map(.name)'
# ["alice","bob"]

# map_values applies to object values
echo '{"a":1,"b":2,"c":3}' | jq 'map_values(. + 10)'
# {"a":11,"b":12,"c":13}

select + map Together

1
2
3
4
5
# Get names of all admin users
jq '[.users[] | select(.role == "admin") | .name]'

# Equivalent using map + select
jq '.users | map(select(.role == "admin")) | map(.name)'

sort_by and group_by

1
2
3
4
5
6
echo '[{"name":"charlie","age":25},{"name":"alice","age":30},{"name":"bob","age":25}]' \
  | jq 'sort_by(.age, .name)'
# [{"name":"bob","age":25},{"name":"charlie","age":25},{"name":"alice","age":30}]

jq 'group_by(.age)'
# [[{"name":"bob","age":25},{"name":"charlie","age":25}],[{"name":"alice","age":30}]]

unique and unique_by

1
2
3
4
5
6
echo '[1,2,2,3,3,3]' | jq 'unique'
# [1,2,3]

echo '[{"id":1,"v":"a"},{"id":1,"v":"b"},{"id":2,"v":"c"}]' \
  | jq 'unique_by(.id)'
# [{"id":1,"v":"a"},{"id":2,"v":"c"}]

min_by and max_by

1
2
3
4
5
6
echo '[{"name":"alice","score":85},{"name":"bob","score":92},{"name":"carol","score":78}]' \
  | jq 'max_by(.score)'
# {"name":"bob","score":92}

jq '[max_by(.score), min_by(.score)] | map(.name)'
# ["bob","carol"]

flatten

1
2
3
4
5
echo '[[1,2],[3,[4,5]]]' | jq 'flatten'
# [1,2,3,4,5]

jq 'flatten(1)'   # One level only
# [1,2,3,[4,5]]

add

Reduces an array by addition — works on numbers, strings, arrays, and objects:

1
2
3
4
5
6
7
8
echo '[1,2,3,4]' | jq 'add'
# 10

echo '["a","b","c"]' | jq 'add'
# "abc"

echo '[{"a":1},{"b":2},{"a":3}]' | jq 'add'
# {"a":3,"b":2}   (later values overwrite earlier ones)

String Operations

String Interpolation — "\(expr)"

1
2
3
echo '{"name":"alice","role":"admin"}' \
  | jq '"User \(.name) has role: \(.role)"'
# "User alice has role: admin"

split and join

1
2
3
4
5
echo '"a,b,c,d"' | jq 'split(",")'
# ["a","b","c","d"]

echo '["a","b","c"]' | jq 'join(", ")'
# "a, b, c"

ltrimstr, rtrimstr, startswith, endswith

1
2
3
4
5
6
7
8
echo '"hello-world"' | jq 'ltrimstr("hello-")'
# "world"

echo '"image:latest"' | jq 'rtrimstr(":latest")'
# "image"

echo '"api/v1/users"' | jq 'startswith("api/")'
# true

ascii_downcase and ascii_upcase

1
2
echo '"Hello World"' | jq 'ascii_downcase'
# "hello world"

test — Regex Matching

1
2
3
4
5
6
7
echo '"alice@example.com"' | jq 'test("^[a-z]+@")'
# true

# Filter array by regex
echo '["alice","bob123","carol"]' \
  | jq '[.[] | select(test("[0-9]"))]'
# ["bob123"]

capture — Named Regex Groups

1
2
echo '"2026-03-25"' | jq 'capture("(?<year>[0-9]{4})-(?<month>[0-9]{2})-(?<day>[0-9]{2})")'
# {"year":"2026","month":"03","day":"25"}

Conditionals and Logic

if-then-else

1
2
3
4
5
6
echo '42' | jq 'if . > 40 then "big" else "small" end'
# "big"

echo '[{"name":"alice","score":85},{"name":"bob","score":45}]' \
  | jq '[.[] | {name, result: (if .score >= 60 then "pass" else "fail" end)}]'
# [{"name":"alice","result":"pass"},{"name":"bob","result":"fail"}]

// — Alternative Operator

Returns the right side if the left is false or null:

1
2
3
4
5
echo '{"name":"alice"}' | jq '.email // "no email"'
# "no email"

echo 'null' | jq '. // "default"'
# "default"

This is the idiomatic way to set defaults for missing fields.

and, or, not

1
2
3
4
5
6
echo '{"active":true,"role":"admin"}' \
  | jq '.active and (.role == "admin")'
# true

jq '.active | not'
# false

reduce — Aggregation

reduce iterates over a stream, accumulating a value:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# Sum all values
echo '[1,2,3,4,5]' | jq 'reduce .[] as $x (0; . + $x)'
# 15

# Build a lookup map from an array
echo '[{"id":"u1","name":"alice"},{"id":"u2","name":"bob"}]' \
  | jq 'reduce .[] as $item ({}; . + {($item.id): $item.name})'
# {"u1":"alice","u2":"bob"}

# Find the maximum
echo '[3,1,4,1,5,9,2,6]' \
  | jq 'reduce .[] as $x (0; if $x > . then $x else . end)'
# 9

path and getpath / setpath — Deep Manipulation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Get the path to a value
echo '{"a":{"b":{"c":42}}}' | jq 'path(.a.b.c)'
# ["a","b","c"]

# Get value at a path
jq 'getpath(["a","b","c"])'
# 42

# Set value at a path
jq 'setpath(["a","b","c"]; 99)'
# {"a":{"b":{"c":99}}}

# Delete a path
jq 'delpaths([["a","b","c"]])'
# {"a":{"b":{}}}

env — Reading Environment Variables

1
2
3
4
5
6
7
export MY_USER="alice"
echo '{}' | jq --arg user "$MY_USER" '{user: $user}'
# {"user":"alice"}

# Or use env object directly
echo '{}' | jq '{user: env.MY_USER}'
# {"user":"alice"}

Passing Values In — --arg, --argjson, --slurpfile

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# Pass a string
jq --arg name "alice" '.users[] | select(.name == $name)'

# Pass a number (argjson parses as JSON)
jq --argjson threshold 80 '.scores[] | select(. > $threshold)'

# Pass a boolean
jq --argjson debug true 'if $debug then . else empty end'

# Pass a file as a JSON value
jq --slurpfile config config.json '.settings + $config[0]'

# Pass a raw string from a file
jq --rawfile template template.txt '{body: $template}'

Output Flags

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Raw output — strip quotes (essential for shell scripting)
echo '"hello world"' | jq -r '.'
# hello world   (no quotes)

# Compact output — single line
echo '{"a":1,"b":2}' | jq -c '.'
# {"a":1,"b":2}

# Raw input — treat each line of input as a string
printf 'alice\nbob\ncarol\n' | jq -R '{"name": .}'
# {"name":"alice"}
# {"name":"bob"}
# {"name":"carol"}

# Slurp — read all input into a single array
echo -e '1\n2\n3' | jq -s '.'
# [1,2,3]

# Null input — don't read stdin
jq -n '{generated: true, date: "2026-03-25"}'

# Tab-indented output
jq --tab '.'

# Indent N spaces
jq --indent 2 '.'

# Join output (no newlines between outputs)
echo '[1,2,3]' | jq -j '.[] | tostring + ","'
# 1,2,3,

Real-World Recipes

AWS CLI

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# List all EC2 instance IDs and their states
aws ec2 describe-instances \
  | jq -r '.Reservations[].Instances[] | "\(.InstanceId)\t\(.State.Name)"'

# Get private IPs of all running instances
aws ec2 describe-instances \
  --filters Name=instance-state-name,Values=running \
  | jq -r '.Reservations[].Instances[].PrivateIpAddress'

# List S3 buckets created after a date
aws s3api list-buckets \
  | jq -r '.Buckets[] | select(.CreationDate > "2025-01-01") | .Name'

# Get all IAM users with their last login
aws iam list-users \
  | jq -r '.Users[] | [.UserName, (.PasswordLastUsed // "never")] | @tsv'

# Find security groups with 0.0.0.0/0 ingress
aws ec2 describe-security-groups \
  | jq -r '.SecurityGroups[] |
      select(.IpPermissions[].IpRanges[]?.CidrIp == "0.0.0.0/0") |
      "\(.GroupId)\t\(.GroupName)"'

Azure CLI

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# List all VMs with their sizes and locations
az vm list \
  | jq -r '.[] | [.name, .location, .hardwareProfile.vmSize] | @tsv'

# Get all resource groups and their tags
az group list \
  | jq -r '.[] | "\(.name): \(.tags // {} | to_entries | map("\(.key)=\(.value)") | join(","))"'

# Find all storage accounts without HTTPS enforcement
az storage account list \
  | jq -r '.[] | select(.enableHttpsTrafficOnly == false) | .name'

# List all AKS clusters and their Kubernetes versions
az aks list \
  | jq -r '.[] | "\(.name)\t\(.kubernetesVersion)\t\(.location)"'

kubectl

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# List all pods and their status
kubectl get pods -A -o json \
  | jq -r '.items[] | [.metadata.namespace, .metadata.name, .status.phase] | @tsv'

# Find pods not in Running state
kubectl get pods -A -o json \
  | jq -r '.items[] | select(.status.phase != "Running") |
      "\(.metadata.namespace)/\(.metadata.name): \(.status.phase)"'

# Get container image versions for a deployment
kubectl get deployment myapp -o json \
  | jq -r '.spec.template.spec.containers[] | "\(.name): \(.image)"'

# List all services with their ClusterIP and ports
kubectl get services -A -o json \
  | jq -r '.items[] | "\(.metadata.namespace)\t\(.metadata.name)\t\(.spec.clusterIP)\t\([.spec.ports[]?.port | tostring] | join(","))"'

# Find all pods using a specific image
kubectl get pods -A -o json \
  | jq -r --arg img "nginx" \
      '.items[] | select(.spec.containers[].image | startswith($img)) |
       "\(.metadata.namespace)/\(.metadata.name)"'

# Get resource requests and limits
kubectl get pods -A -o json | jq -r '
  .items[] |
  .metadata.name as $pod |
  .spec.containers[] |
  "\($pod)\t\(.name)\t\(.resources.requests.cpu // "-")\t\(.resources.requests.memory // "-")"'

GitHub API / REST APIs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Get open PRs with author and title
curl -s "https://api.github.com/repos/owner/repo/pulls" \
  | jq -r '.[] | "\(.number)\t\(.user.login)\t\(.title)"'

# Count open issues by label
curl -s "https://api.github.com/repos/owner/repo/issues" \
  | jq 'group_by(.labels[0].name) | map({label: .[0].labels[0].name, count: length})'

# Get release asset download URLs
curl -s "https://api.github.com/repos/owner/repo/releases/latest" \
  | jq -r '.assets[] | "\(.name)\t\(.browser_download_url)"'

Docker

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# List containers with name, image, and status
docker inspect $(docker ps -q) \
  | jq -r '.[] | "\(.Name)\t\(.Config.Image)\t\(.State.Status)"'

# Get exposed ports for all running containers
docker inspect $(docker ps -q) \
  | jq -r '.[] | .Name as $name |
      .NetworkSettings.Ports | to_entries[] |
      select(.value != null) |
      "\($name): \(.key) -> \(.value[0].HostPort)"'

# Find containers with no resource limits set
docker inspect $(docker ps -q) \
  | jq -r '.[] | select(.HostConfig.Memory == 0) | .Name'

Processing Multiple Files

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Process all JSON files in a directory
jq '.name' *.json

# Slurp multiple files into one array
jq -s '.' file1.json file2.json file3.json

# Process files and include filename context
for f in configs/*.json; do
    jq -r --arg file "$f" '"\($file): \(.version)"' "$f"
done

Streaming Large Files

For very large JSON files that don’t fit in memory, use --stream:

1
2
3
4
5
# Stream a large file — outputs path/value pairs
jq --stream 'select(length == 2)' large.json | head -20

# Extract just the top-level keys without loading the whole thing
jq -n --stream 'first(inputs | select(length == 2 and .[0][0] == "users") | .[1])'

@format Strings — Output Encoding

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# TSV output (great for pasting into spreadsheets)
echo '[{"name":"alice","age":30},{"name":"bob","age":25}]' \
  | jq -r '.[] | [.name, .age] | @tsv'
# alice   30
# bob     25

# CSV output
jq -r '.[] | [.name, .age] | @csv'
# "alice",30
# "bob",25

# Base64 encode
echo '"hello world"' | jq -r '@base64'
# aGVsbG8gd29ybGQ=

# Base64 decode
echo '"aGVsbG8gd29ybGQ="' | jq -r '@base64d'
# hello world

# URL encode
echo '"hello world & more"' | jq -r '@uri'
# hello%20world%20%26%20more

# HTML encode
echo '"<script>alert(1)</script>"' | jq -r '@html'
# &lt;script&gt;alert(1)&lt;/script&gt;

# sh — shell-safe quoting
echo '"user input; rm -rf /"' | jq -r '@sh'
# 'user input; rm -rf /'

Using @sh Safely in Scripts

When passing jq output to shell commands, always use @sh or -r with proper quoting:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# UNSAFE — breaks on spaces/special chars
name=$(echo "$json" | jq '.name')
echo $name

# SAFE — use -r and quote the variable
name=$(echo "$json" | jq -r '.name')
echo "$name"

# SAFE — use @sh for values that go into eval or complex commands
eval "$(echo "$json" | jq -r '@sh "name=\(.name) age=\(.age)"')"

Defining Functions

1
2
3
4
5
6
7
jq '
  def double: . * 2;
  def clamp(lo; hi): if . < lo then lo elif . > hi then hi else . end;

  [1,2,3,4,5] | map(double) | map(clamp(4; 8))
'
# [4,4,6,8,8]

Functions can be stored in a file (~/.jq or a .jq file) and included:

1
2
3
4
5
6
7
8
# ~/.jq
def to_gib: . / 1073741824 | (. * 10 | round) / 10;
def humanize_bytes:
  if . >= 1073741824 then "\(to_gib) GiB"
  elif . >= 1048576 then "\(. / 1048576 | round) MiB"
  elif . >= 1024 then "\(. / 1024 | round) KiB"
  else "\(.) B"
  end;
1
2
3
4
5
echo '1073741824' | jq 'humanize_bytes'
# "1 GiB"

echo '5368709120' | jq 'humanize_bytes'
# "5 GiB"

Practical Script Patterns

Generate a Config File from JSON

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
#!/usr/bin/env bash
# Generate nginx upstream block from service registry
services=$(curl -s http://registry/services)

echo "$services" | jq -r '
  .[] |
  "upstream \(.name) {",
  (.instances[] | "    server \(.host):\(.port);"),
  "}"
'

Validate JSON Structure

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
validate_config() {
    local file="$1"
    local errors=0

    # Check required fields exist
    for field in name version environment; do
        if ! jq -e ".$field" "$file" > /dev/null 2>&1; then
            echo "ERROR: Missing required field: $field" >&2
            ((errors++))
        fi
    done

    # Check environment is valid
    env=$(jq -r '.environment' "$file")
    if [[ ! "$env" =~ ^(dev|staging|production)$ ]]; then
        echo "ERROR: Invalid environment: $env" >&2
        ((errors++))
    fi

    return $errors
}

Merge JSON Objects

1
2
3
4
5
# Merge two JSON files, second wins on conflicts
jq -s '.[0] * .[1]' base.json overrides.json

# Deep merge all JSON files in a directory
jq -s 'reduce .[] as $x ({}; . * $x)' configs/*.json

Patch a JSON File In-Place

1
2
3
4
5
6
# Update a field in a JSON file
tmp=$(mktemp)
jq '.version = "2.0.1"' package.json > "$tmp" && mv "$tmp" package.json

# Add an element to an array
jq '.dependencies += ["newpkg"]' config.json | sponge config.json

Transform a Log Stream

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Pretty-print a stream of newline-delimited JSON logs (NDJSON)
tail -f app.log | jq -c 'select(.level == "error") | {ts: .timestamp, msg: .message}'

# Count errors by service in the last 1000 lines
tail -1000 app.log | jq -rs '
  [.[] | select(.level == "error")] |
  group_by(.service) |
  map({service: .[0].service, count: length}) |
  sort_by(.count) | reverse[]
'

Quick Reference

.                   Identity (pretty-print)
.foo                Field access
.foo.bar            Nested field
.foo?               Optional field (no error if missing)
.[0]                Array index
.[-1]               Last element
.[2:5]              Slice
.[]                 Iterate array/object

|                   Pipe
,                   Produce multiple outputs
( expr )            Grouping

[expr]              Collect into array
{key: expr}         Build object
{name}              Shorthand: {name: .name}

select(bool)        Filter — pass through if true
map(f)              Apply f to each element
map_values(f)       Apply f to each object value

length              Count
keys / values       Object keys / values
has("key")          Check field existence
type                Get type as string
to_entries          [{key,value}] pairs
from_entries        Reverse of to_entries
with_entries(f)     map over {key,value} pairs

sort / sort_by(f)   Sort
group_by(f)         Group into arrays
unique / unique_by  Deduplicate
flatten             Flatten nested arrays
add                 Sum / concatenate array
first / last        First / last of stream
min_by / max_by     Extremes

if C then A else B  Conditional
. // default        Alternative (null/false fallback)
not                 Boolean negation

reduce .[] as $x (init; expr)    Fold
@tsv / @csv / @base64 / @uri / @sh    Output formats

-r    Raw string output
-c    Compact output
-n    No input
-s    Slurp into array
-R    Raw string input
--arg name val      Bind string variable
--argjson name val  Bind JSON variable

jq rewards investment. The first twenty minutes of learning pays off every time you reach for it instead of opening Python for a one-liner. The next step is keeping ~/.jq stocked with your own helper functions — once you’ve written humanize_bytes or to_epoch once, you never write it again.

Comments