Using jq to concatenate directory of JSON files

Question

I have a directory of about 100 JSON files, each an array of 100 simple records, that I want to concatenate into one file for inclusion as static data in an app, so I don't have to make repeated API calls to retrieve small pieces. (I'm limited to downloading only 100 records at a time; that's why I have 100 short files.)

Here's a sample file, shortened to two records for display here:

[
 {
   "id": 11531,
   "title": "category 1",
   "count": 5
 },
 {
   "id": 11532,
   "title": "category 2",
   "count": 5
 }
]

My research led to a solution that works but only for two files with two records each:

jq -s '.[0] + .[1]' file1.json file2.json > output.json

This source also suggested this line would work to handle a directory (right now only two files in it):

jq -s 'reduce .[] as $item ({}; . * $item)' json_files/* > output.json

but I get an error:

jq: error (at json_files/categories-11-20.json:0): object ({}) and array ([{"id":1153...) cannot be multiplied

I thought maybe the problem was the * trying to multiply, so I tried + in that place, but I get a ... cannot be added. message.

Is there a way to do this through jq or is there a better tool?

Answer 1

The simplest and perfectly reasonable approach would be to use the -s command-line option and add along the following lines:

jq -s add json_files/* 

Of course you may wish to specify the list of files differently. The order in which they are specified is also significant.

Notes:

• This Q is really just a variant of Use jq to concatenate JSON arrays in multiple files

• reduce can also be used, but you would need to start with null or [] rather than {} .

• The operator '*' is (not surprisingly) quite different from '+'!