Using jq to combine multiple JSON files

Question

First off, I am not an expert with JSON files or with JQ. But here's my problem:

I am simply trying to download to card data (for the MtG card game) through an API, so I can use it in my own spreadsheets etc.

The card data from the API comes in pages, since there is so much of it, and I am trying to find a nice command line method in Windows to combine the files into one. That will make it nice and easy for me to use the information as external data in my workbooks.

The data from the API looks like this:

{
  "object": "list",
  "total_cards": 290,
  "has_more": true,
  "next_page": "https://api.scryfall.com/cards/search?format=json&include_extras=false&order=set&page=2&q=e%3Alea&unique=cards",
  "data": [
    {
      "object": "card",
      "id": "d5c83259-9b90-47c2-b48e-c7d78519e792",
      "oracle_id": "c7a6a165-b709-46e0-ae42-6f69a17c0621",
      "multiverse_ids": [
        232
      ],
      "name": "Animate Wall",
      ......
    }
    {
      "object": "card",
      ......
    }
  ]
}

Basically I need to take what's inside the "data" part from each file after the first, and merge it into the first file.

I have tried a few examples I found online using jq, but I can't get it to work. I think it might be because in this case the data is sort of under an extra level, since there is some basic information, then the "data" category is beneath it. I don't know.

Anyway, any help on how to get this going would be appreciated. I don't know much about this, but I can learn quickly so even any pointers would be great.

Thanks!

Answer 1

To merge the .data elements of all the responses into the first response, you could run:

jq 'reduce inputs.data as $s (.; .data += $s)' page1.json page2.json ...

Alternatives

You could use the following filter in conjunction with the -n command-line option:

reduce inputs as $s (input; .data += ($s.data))

Or if you simply want an object of the form {"data": [ ... ]} then (again assuming you invoke jq with the -n command-line option) the following jq filter would suffice:

{data: [inputs.data] | add}

Answer 2

Just to provide closure, @peak provided the solution. I am using it in conjunction with the method found here for using wildcards in batch files to address multiple files. The code looks like this now:

set expanded_list=
for /f "tokens=*" %%F in ('dir /b /a:-d "All Cards\!setname!_*.json"') do call set expanded_list=!expanded_list! "All Cards\%%F"
jq-win32 "reduce inputs.data as $s (.; .data += $s)" !expanded_list! > "All Cards\!setname!.json"

All the individual pages for each card set are named "setname"_"pagenumber".json

The code finds all the pages for each set and combines them into one variable which I can pass into jq.

Thanks again!