NiFi：JoltTransformJSON 规范

Question

I have following JSON:

{
  "results": [
    {
      "customerClient": {
        "resourceName": "customers/7876562723/customerClients/8506630423",
        "clientCustomer": "customers/8506630423",
        "hidden": false,
        "level": "1",
        "manager": false,
        "descriptiveName": "BMW",
        "id": "85061423"
      }
    },
    {
      "customerClient": {
        "resourceName": "customers/7876562723/customerClients/6736523142",
        "clientCustomer": "customers/6736523142",
        "hidden": false,
        "level": "1",
        "manager": true,
        "descriptiveName": "Mercedes",
        "id": "67363142"
      }
    }
}
  ],
  "fieldMask": "customerClient.clientCustomer,customerClient.hidden,customerClient.level,customerClient.descriptiveName,customerClient.id,customerClient.manager"
}

What i want:

1. remove fieldMask parameter, so i don't need results as array.
2. remove results and customerCient . So my JSON should look like this:

{
   "resourceName": "customers/7876562723/customerClients/8506630423",
   "clientCustomer": "customers/8506630423",
   "hidden": false,
   "level": "1",
   "manager": false,
   "descriptiveName": "BMW",
   "id": "85061423"
},
{
   "resourceName": "customers/7876562723/customerClients/6736523142",
   "clientCustomer": "customers/6736523142",
   "hidden": false,
   "level": "1",
   "manager": true,
   "descriptiveName": "Mercedes",
   "id": "67363142"
}

To remove results i'm using SplitJson with split value: $.results . And now i'm stuck at modifying it with JOLT specification. How can I do described above actions with only JoltTransformJSON ?

Answer 1

To start with - you json is wrong, you have an extra }.

Not Using jolt

1 - generate data

2 - evaluatejson with $.results.[*].customerClient as attribute results

3 - replacetext with ${results:replace(']',''):replace('[','')}

this will giive you

{
    "resourceName": "customers/7876562723/customerClients/8506630423",
    "clientCustomer": "customers/8506630423",
    "hidden": false,
    "level": "1",
    "manager": false,
    "descriptiveName": "BMW",
    "id": "85061423"
},
{
    "resourceName": "customers/7876562723/customerClients/6736523142",
    "clientCustomer": "customers/6736523142",
    "hidden": false,
    "level": "1",
    "manager": true,
    "descriptiveName": "Mercedes",
    "id": "67363142"
}

Answer 2

Ok, so I believe JOLT can only output one object, so transforming each element of the array to unique objects wouldn't be possible in one JOLT. However, you can get most of the way there without putting FlowFile data in to Attributes.

We can:

• delete fieldMask
• Remove the results array level

Use this to test out JOLTs: https://jolt-demo.appspot.com/#inception

Assuming your JSON is actually valid, so:

{
  "results": [
    {
      "customerClient": {
        "resourceName": "customers/7876562723/customerClients/8506630423",
        "clientCustomer": "customers/8506630423",
        "hidden": false,
        "level": "1",
        "manager": false,
        "descriptiveName": "BMW",
        "id": "85061423"
      }
    },
    {
      "customerClient": {
        "resourceName": "customers/7876562723/customerClients/6736523142",
        "clientCustomer": "customers/6736523142",
        "hidden": false,
        "level": "1",
        "manager": true,
        "descriptiveName": "Mercedes",
        "id": "67363142"
      }
    }
  ],
  "fieldMask": "customerClient.clientCustomer,customerClient.hidden,customerClient.level,customerClient.descriptiveName,customerClient.id,customerClient.manager"
}

You could use this JOLT spec:

[
  {
    "operation": "remove",
    "spec": {
      "fieldMask": ""
    }
  },
  {
    "operation": "shift",
    "spec": {
      "results": {
        "*": {
          "customerClient": "customerClient-&1"
        }
      }
    }
  }
]

Which gives you following result:

{
  "customerClient-0" : {
    "resourceName" : "customers/7876562723/customerClients/8506630423",
    "clientCustomer" : "customers/8506630423",
    "hidden" : false,
    "level" : "1",
    "manager" : false,
    "descriptiveName" : "BMW",
    "id" : "85061423"
  },
  "customerClient-1" : {
    "resourceName" : "customers/7876562723/customerClients/6736523142",
    "clientCustomer" : "customers/6736523142",
    "hidden" : false,
    "level" : "1",
    "manager" : true,
    "descriptiveName" : "Mercedes",
    "id" : "67363142"
  }
}

So you can transform your JSON to a flat structure using just the JOLT, then you could use SplitJSON to break each object up (if needed).

You should consider using Records instead of SplitJSON, this would probably be more efficient.

Read about records:

• https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi
• https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apache.nifi.processors.standard.UpdateRecord/index.html
• https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.12.1/org.apache.nifi.json.JsonTreeReader/index.html

Another interesting alternative could be ScriptedTransformRecord