jq：按嵌套结构分组并展平JSON

Question

I'm new to jq and command line tools in general, but I need to group by nested structures in a JSON file and flatten the nested structures, and I've not been able to find a workable solution for a few days, here's a sample of my JSON.

[
  {
    "Value1": "0",
    "Conversions": "0",
    "Revenue": "0.00",
    "serverTimestamp": 84615198,
    "pluginsIcons": [
      {
        "pluginName": "pdf",
        "pluginIcon": "pdf1"
      },
      {
        "pluginName": "java",
        "pluginIcon": "java1"
      }
    ],
    "plugins": "pdf, java",
    "customVariables": {
      "3": {
        "customVariableValue3": "F",
        "customVariableName3": "Gender"
      },
      "2": {
        "customVariableValue2": "Person",
        "customVariableName2": "Role"
      },
      "1": {
        "customVariableValue1": "Partner1",
        "customVariableName1": "Partner"
      }
    },
    "interactions": "7",
    "actions": "3",
    "actionDetails": [
      {
        "timestamp": 84615195,
        "interactionPosition": "1",
        "type": "action"
      },
      {
        "timestamp": 84615145,
        "interactionPosition": "2",
        "type": "action"
      },
      {
        "timestamp": 84615693,
        "interactionPosition": "3",
        "type": "action",
        "customVariables": {
          "2": {
            "customVariablePageValue2": "value2",
            "customVariablePageName2": "name2"
          },
          "1": {
            "customVariablePageValue1": "value1",
            "customVariablePageName1": "name1"
          }
        }
      }
    ],
    "operatingSystem": "Windows 10"
  },
  {
    "Value1": "18",
    "Conversions": "1",
    "Revenue": "0.00",
    "serverTimestamp": 84615189,
    "pluginsIcons": [
      {
        "pluginName": "pdf",
        "pluginIcon": "pdf1"
      }
    ],
    "plugins": "pdf",
    "customVariables": {
      "3": {
        "customVariableValue3": "M",
        "customVariableName3": "Gender"
      },
      "2": {
        "customVariableValue2": "Admin",
        "customVariableName2": "Role"
     },
      "1": {
        "customVariableValue1": "Place",
        "customVariableName1": "Subdomain"
      }
    },
    "interactions": "6",
    "actions": "3",
    "actionDetails": [
      {
        "timestamp": 84635189,
        "timeSpent": "11",
        "interactionPosition": "1",
        "type": "action"
      },
      {
        "timestamp": 846351834,
        "timeSpent": "11",
        "interactionPosition": "2",
        "type": "search"
      },
      {
        "timestamp": 846351832,
        "timeSpent": "1",
        "interactionPosition": "3",
        "type": "action",
        "customVariables": {
          "2": {
            "customVariablePageValue2": "value2",
            "customVariablePageName2": "name2"
          },
          "1": {
            "customVariablePageValue3": "value3",
            "customVariablePageName3": "name3"
          }
        },
        "generationTime": "890"
      }
    ],
    "operatingSystem": "Windows 10"
  }
]

The way it should look at the end result would be with one flattened entry for each "action" in the nested arrays under "actionDetails"

I have been able to flatten the structures, but then grouping by (and duplicating the other columns for each action) becomes convoluted. Grouping by the "action"s before flattening has not worked for me because they're nested.

An example of how the first entry in the original JSON should look afterwards is:

[
  {
    "timestamp": 84615195,
    "interactionPosition": "1",
    "type": "action",
    "Value1": "0",
    "Conversions": "0",
    "Revenue": "0.00",
    "pluginName1": "pdf",
    "pluginIcon1": "pdf",
    "pluginName2": "java",
    "pluginIcon2": "java",
    "plugins": "pdf, java",
    "Gender": "F",
    "Role": "Person",
    "Partner": "Partner1",
    "interactions": "7",
    "actions": "3",
    "operatingSystem": "Windows 10"
  },
  {
    "timestamp": 84615145,
    "interactionPosition": "2",
    "type": "action",
    "Value1": "0",
    "Conversions": "0",
    "Revenue": "0.00",
    "pluginName1": "pdf",
    "pluginIcon1": "pdf",
    "pluginName2": "java",
    "pluginIcon2": "java",
    "plugins": "pdf, java",
    "Gender": "F",
    "Role": "Person",
    "Partner": "Partner1",
    "interactions": "7",
    "actions": "3",
    "operatingSystem": "Windows 10"
  },
  {
    "timestamp": 84615693,
    "interactionPosition": "3",
    "type": "action",
    "Value1": "0",
    "Conversions": "0",
    "Revenue": "0.00",
    "pluginName1": "pdf",
    "pluginIcon1": "pdf",
    "pluginName2": "java",
    "pluginIcon2": "java",
    "plugins": "pdf, java",
    "Gender": "F",
    "Role": "Person",
    "Partner": "Partner1",
    "interactions": "7",
    "actions": "3",
    "operatingSystem": "Windows 10",
    "name1": "value1",
    "name2": "value2"
   }
]

You may note in the above that some of the flattened key names have been replaced by an associated value (inside the same nested structure). This isn't completely necessary, but it would be a nice bonus. Also worth noting: My JSON is large (800MB), and I would like to do that, but I suppose that point would be best posed in another question.

Thanks in advance for any assistance or advice!

Answer 1

The following answer does not deal with every requirement you've mentioned but it will hopefully get you over the main hurdle you've evidently been facing.

Since your requirements regarding the "customVariables" are not clear to me, I'll ignore .customVariables completely, and hopefully you'll also be able to handle .pluginsIcons yourself once you're over the main hurdle. So for clarity, I'll simply delete those keys.

As I understand it, you want some grouping to take place after the flattening based on .actionDetails. These requirements are also unclear to me so let's focus on the flattening:

.[]
| .actionDetails[] + (del(.actionDetails) | del(.customVariables) | del(.pluginsIcons))

This produces a stream of JSON objects, the first two of which are:

{
  "timestamp": 84615195,
  "interactionPosition": "1",
  "type": "action",
  "Value1": "0",
  "Conversions": "0",
  "Revenue": "0.00",
  "serverTimestamp": 84615198,
  "plugins": "pdf, java",
  "interactions": "7",
  "actions": "3",
  "operatingSystem": "Windows 10"
}
{
  "timestamp": 84615145,
  "interactionPosition": "2",
  "type": "action",
  "Value1": "0",
  "Conversions": "0",
  "Revenue": "0.00",
  "serverTimestamp": 84615198,
  "plugins": "pdf, java",
  "interactions": "7",
  "actions": "3",
  "operatingSystem": "Windows 10"
}

This is very similar to the expected output you've shown, so hopefully you can take it from here.