用 jq 导出 JSON 到 CSV

Question

I have the following JSON file:

{"data":[{"id":"DDM003","base":[{"date":"2020-06-04T00:30:00Z","value":335.2,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":330.1,"state":"A","validated":2}]},{"id":"DTR001","base":[{"date":"2020-06-04T00:30:00Z","value":0.2,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":0.1,"state":"A","validated":2}]},{"id":"FFM003","base":[{"date":"2020-06-04T00:30:00Z","value":2.62,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":3.15,"state":"A","validated":2}]},{"id":"RAIN12","base":[{"date":"2020-06-04T00:30:00Z","value":15.0,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":16.0,"state":"A","validated":2}]},{"id":"RHM003","base":[{"date":"2020-06-04T00:30:00Z","value":85.41,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":85.35,"state":"A","validated":2}]},{"id":"WVM003","base":[{"date":"2020-06-04T00:30:00Z","value":2.56,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":3.08,"state":"A","validated":2}]},{"id":"TLR001","base":[{"date":"2020-06-04T00:30:00Z","value":14.28,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":14.36,"state":"A","validated":2}]},{"id":"THR001","base":[{"date":"2020-06-04T00:30:00Z","value":14.07,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":14.23,"state":"A","validated":2}]},{"id":"PPR001","base":[{"date":"2020-06-04T00:30:00Z","value":999.2,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":998.9,"state":"A","validated":2}]},{"id":"RHR001","base":[{"date":"2020-06-04T00:30:00Z","value":80.5,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":80.0,"state":"A","validated":2}]},{"id":"WDR001","base":[{"date":"2020-06-04T00:30:00Z","value":317.71,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":320.31,"state":"A","validated":2}]},{"id":"WVR001","base":[{"date":"2020-06-04T00:30:00Z","value":2.75,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":2.33,"state":"A","validated":2}]},{"id":"WSR001","base":[{"date":"2020-06-04T00:30:00Z","value":2.91,"state":"A","validated":2},{"date":"2020-06-04T01:00:00Z","value":2.44,"state":"A","validated":2}]}]}

I would like to export it (and any file of this type with a larger date range) into the following CSV format using jq (note the reordering of the fields):

date;WDR001;WVR001;WSR001;TLR001;THR001;DTR001;PPR001;RHR001;DDM003;WVM003;FFM003;RHM003;RAIN12
2020-06-04 00:30:00;317.71;2.75;2.91;14.28;14.36;0.2;999.2;80.5;335.2;2.56;2.62;85.41;15
2020-06-04 01:00:00;320.31;2.33;2.44;14.07;14.23;0.1;998.9;80;330.1;3.08;3.15;85.35;16

with the conditions that any value associated with a "state" attribute (which always exists) different from {A,R,O,W,K} or a "validated" attribute (which can be missing) different from 2 is set to the default value -9999.

Could somebody please help me with the jq filter needed to achieve this?

Many thanks.

Answer 1

Using the -r command-line option, the following filter produces CSV as shown below:

def adjust:
  if (.state | test("^[AROWK]$") | not) or .validated != 2 
  then .value = -9999 
  else . 
  end;


["WDR001", "WVR001", "WSR001", "TLR001", "THR001", "DTR001", "PPR001", "RHR001", "DDM003", "WVM003", "FFM003", "RHM003", "RAIN12"] as $keys
| [.data[] | .id as $id | .base[] | . + {id: $id}]
| group_by(.date)[]
| map(adjust)
| .[0].date as $date
| ( INDEX(.[]; .id ) | map_values({value}) )
| [ $date, ($keys[] as $k | .[$k].value)]
| @csv

In words: store the.id in the.base objects; group these objects by.date; make the adjustment of.value; create a mapping from.id to.value; and finally emit the desired output.

Output:

"2020-06-04T00:30:00Z",317.71,2.75,2.91,14.28,14.07,0.2,999.2,80.5,335.2,2.56,2.62,85.41,15
"2020-06-04T01:00:00Z",320.31,2.33,2.44,14.36,14.23,0.1,998.9,80,330.1,3.08,3.15,85.35,16

It's easy enough to add the headers.

If you want semi-colon separated values without the quotation marks around the date, you could use join(;) instead of @csv .

Terser still

["WDR001", "WVR001", "WSR001", "TLR001", "THR001", "DTR001", "PPR001", "RHR001", "DDM003", "WVM003", "FFM003", "RHM003", "RAIN12"] as $keys
| [.data[] | .base[] + {id}]
| group_by(.date)[]
| map(adjust)
| .[0].date as $date
| ( INDEX(.[]; .id ) | map_values({value}) )
| [ $date, (.[$keys[]]|.value)]
| @csv