Import Google API JSON file to Elasticsearch

Question

I'm totally new with ELK stack and especially with ES. I'm trying to import a JSON file which I got using Google Admin SDK API and I want to import it to Elasticsearch.

So far this is the JSON structure of my data:

{
"kind": "reports#activities",
"nextPageToken": string,
"items": [
{
"kind": "audit#activity",
  "id": {
    "time": datetime,
    "uniqueQualifier": long,
    "applicationName": string,
    "customerId": string
  },
  "actor": {
    "callerType": string,
    "email": string,
    "profileId": long,
    "key": string
  },
  "ownerDomain": string,
  "ipAddress": string,
  "events": [
    {
      "type": string,
      "name": string,
      "parameters": [
        {
          "name": string,
          "value": string,
          "intValue": long,
          "boolValue": boolean
        }
       ]
     }
   ]
  }
 ]
}

So I decided to first use this command to upload the JSON file into ES :

curl -s -XPOST 'localhost:9200/_bulk' --data-binary @documents.json

But I get some errors :

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [START_ARRAY]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [START_ARRAY]"},"status":400}

What should I do ?

Thank you for your help !

Answer 1

That JSON seems to be defining your document structure, so you first need to create an index with a mapping that will match that structure. In your case, you could do it like this:

curl -XPUT localhost:9200/reports -d '{
  "nextPageToken": {
    "type": "string"
  },
  "items": {
    "properties": {
      "kind": {
        "type": "string"
      },
      "id": {
        "properties": {
          "time": {
            "type": "date",
            "format": "date_time"
          },
          "uniqueQualifier": {
            "type": "long"
          },
          "applicationName": {
            "type": "string"
          },
          "customerId": {
            "type": "string"
          }
        }
      },
      "actor": {
        "properties": {
          "callerType": {
            "type": "string"
          },
          "email": {
            "type": "string"
          },
          "profileId": {
            "type": "long"
          },
          "key": {
            "type": "string"
          }
        }
      },
      "ownerDomain": {
        "type": "string"
      },
      "ipAddress": {
        "type": "string"
      },
      "events": {
        "properties": {
          "type": {
            "type": "string"
          },
          "name": {
            "type": "string"
          },
          "parameters": {
            "properties": {
              "name": {
                "type": "string"
              },
              "value": {
                "type": "string"
              },
              "intValue": {
                "type": "long"
              },
              "boolValue": {
                "type": "boolean"
              }
            }
          }
        }
      }
    }
  }
}'

This being done, you can now index your reports#activities documents that follow the above structure using a bulk call. The syntax of bulk calls is precisely defined here , ie you need a command line (what to do) followed on the next line by the document source (what to index) which must not contain any new lines!

So, you need to reformat your documents.json file like this (make sure to add a new line after the second line). Also note that I've added some dummy data to illustrate the process:

{"index": {"_index": "reports", "_type": "activity"}}
{"kind":"reports#activities","nextPageToken":"string","items":[{"kind":"audit#activity","id":{"time":"2016-05-31T00:00:00.000Z","uniqueQualifier":1,"applicationName":"string","customerId":"string"},"actor":{"callerType":"string","email":"string","profileId":1,"key":"string"},"ownerDomain":"string","ipAddress":"string","events":[{"type":"string","name":"string","parameters":[{"name":"string","value":"string","intValue":1,"boolValue":true}]}]}]}