Shopping List

Basics

We will assume you are familiar with JSON syntax and data types.

jq works by passing the incoming JSON data through a single expression (written as a pipeline of filters) to achieve the desired transformed data.

Filters and Pipes

Filters are also known as Expressions.

A filter takes an input and produces an output. Like the way you work in a unix shell, you can join filters with a pipe | to connect the output of one filter to the input of another.

The Identity Filter: .

This is the simplest filter. It simply passes its input to its output.

$ echo '[1, 2, 3]' | jq '.'
[
  1,
  2,
  3
]

Arrays

This will be quick introduction to working with arrays. We will cover this topic in greater detail later.

Array elements are accessed with brackets, and are zero-indexed.

$ echo '[10, 20, 30]' | jq '.[1]'
20

A filter can build an array by wrapping an expression in [ and ]

• with a known list of elements:

  jq -n '[1, 2, 3]'
  

• to collect a stream of elements: for example, range is a function that outputs a stream of numbers

  $ jq -n 'range(10; 70; 15)'
  10
  25
  40
  55
  

  Using [] collects the results of the expression into an array

  $ jq -c -n '[range(10; 70; 15)]'
  [10,25,40,55]
  

Comma is an operator

The comma is not just syntax that separates array elements. Comma is an operator that joins streams.

For example [1, 2, 3] is a filter that uses the array constructor [] to collect the result of joining the three expressions 1, 2 and 3.

Did you notice the semi-colons in range(10; 70; 15) above? Because commas have a specific purpose in the jq language, functions that take multiple arguments use semi-colons to separate the arguments.

Objects

A quick introduction to objects.

Similar to many programming languages, use dots to access object properties

$ echo '{"foo": {"bar": "qux"}}' | jq '.foo.bar'
"qux"

$ echo '{"foo bar": "qux"}' | jq '.["foo bar"]'
"qux"

You can construct an object with {} and key: value pairs. Quotes are not required around keys that are "simple" strings.

jq -n '{question: (6 * 9), answer: 42}'

outputs

{
  "question": 54,
  "answer": 42
}

To treat the key as an expression, you must wrap it in parentheses (the following also outputs the same as above).

echo '[{"key":"question", "value":54}, {"key":"answer", "value":42}]' \
| jq '{(.[0].key): .[0].value, (.[1].key): .[1].value}'

Pipelines

For example, given file.json containing

{
  "key1": "value1",
  "key2": [5, 15, 25]
}

Let's calculate the length of the key2 array:

$ jq '.key2 | length' file.json
3

We're piping the output of the .key2 expression as the input to length which unsurprisingly outputs the number of elements in the array.

Filters can ignore their input

In this example, the input JSON data is ignored and has no impact on the output:

$ echo '{"answer": 42}' | jq '6 * 9'
54

Filters can output streams of data

A filter can output more than one value. For example, the .[] filter outputs each element of an array as a separate value:

$ jq -n -c '[1, 2, 3]'
[1,2,3]

$ jq -n -c '[1, 2, 3] | .[]'
1
2
3

Piping such a filter into another will execute the 2nd filter for each value:

$ jq -n -c '[1, 2, 3] | .[] | . * 2'
2
4
6

This is like implicit iteration. Once you understand this technique, you'll realize very powerful jq filters can be very concise.

Parentheses

Parentheses are used to group sub-expressions together to enforce the order of operations, just like in other languages. In jq, the need for them can appear to be somewhat surprising.

For example, let's say we want to construct an array with 2 elements: the square root of 9; and e raised to the power 1. The two individual expressions are 9 | sqrt and 1 | exp. We expect the output to be the array [3, 2.7...]

$ jq -n '[ 9|sqrt, 1|exp ]'
[
  20.085536923187668,
  2.718281828459045
]

Why didn't we get what we expected? jq interprets that like this:

[ ((9|sqrt), 1) | exp ]

jq builds a stream of two elements (3 and 1) which are each given to exp.

We need to ensure that exp only takes one number as input. In other words, we need to enforce that the pipe is evaluated before the comma.

$ jq -n '[ 9|sqrt, (1|exp) ]'
[
  3,
  2.718281828459045
]

Functions and Operators

Without going into great depth (functions will be a topic for another exercise), here are some useful builtin functions:

• length

  Given an array as input, output the number of elements in the array.

  $ jq -n '[10, 20, 30, 40] | length'
  4
  

• +

  This operator does different things depending on the type of its operands: it adds numbers, it concatenates strings, it appends arrays, it merges objects.

  $ jq -c -n '
      3 + 4,
      "foo" + "bar",
      ["a", "b"] + ["c"],
      {"m": 10} + {"n": 20}
  '
  7
  "foobar"
  ["a","b","c"]
  {"m":10,"n":20}
  

  add is a function that takes an array and returns an item with all the elements added together using the rules of +. [1, 2, 3] | add outputs 6.

• map

  Given an array as input and a filter as an argument, output an array where the filter is applied to each element

  $ jq -c -n '[10, 20, 30, 40] | map(. / 5)'
  [2,4,6,8]
  

• select

  Given some input and a filter as an argument:

  • if the filter applied to the argument results in a true value, output the input unchanged
  • otherwise, output nothing (not the null value, truly no output)

  For example, given some numbers, select the ones divisible by 3

  $ jq -n 'range(10) | select(. % 3 == 0)'
  0
  3
  6
  9
  

  Recall that range outputs a stream of numbers. select will be invoked once per each number. Only the numbers "passing" the expression are output.

  You often need to select elements of an array. There are a couple of ways to do this.

  With the input ["Anne", "Bob", "Cathy", "Dave"], select the names having length 4.

  • use map and select together

    map(select(length == 4))
    

  • explode the array into elements, select on that stream, and collect the results

    [ .[] | select(length == 4) ]
    

Comments

Comments start with a # character and continue to the end of the line.