將Google API JSON文件導入到Elasticsearch

Question

我對ELK堆棧特別是ES完全陌生。 我正在嘗試導入使用Google Admin SDK API獲得的JSON文件，並且想將其導入到Elasticsearch。

到目前為止，這是我的數據的JSON結構：

{
"kind": "reports#activities",
"nextPageToken": string,
"items": [
{
"kind": "audit#activity",
  "id": {
    "time": datetime,
    "uniqueQualifier": long,
    "applicationName": string,
    "customerId": string
  },
  "actor": {
    "callerType": string,
    "email": string,
    "profileId": long,
    "key": string
  },
  "ownerDomain": string,
  "ipAddress": string,
  "events": [
    {
      "type": string,
      "name": string,
      "parameters": [
        {
          "name": string,
          "value": string,
          "intValue": long,
          "boolValue": boolean
        }
       ]
     }
   ]
  }
 ]
}

因此，我決定首先使用此命令將JSON文件上傳到ES中：

curl -s -XPOST 'localhost:9200/_bulk' --data-binary @documents.json

但是我得到一些錯誤：

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [START_ARRAY]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [START_ARRAY]"},"status":400}

我該怎么辦 ？

謝謝您的幫助 ！

Answer 1

該JSON似乎正在定義您的文檔結構，因此您首先需要創建一個具有匹配該結構的映射的索引。 在您的情況下，您可以這樣做：

curl -XPUT localhost:9200/reports -d '{
  "nextPageToken": {
    "type": "string"
  },
  "items": {
    "properties": {
      "kind": {
        "type": "string"
      },
      "id": {
        "properties": {
          "time": {
            "type": "date",
            "format": "date_time"
          },
          "uniqueQualifier": {
            "type": "long"
          },
          "applicationName": {
            "type": "string"
          },
          "customerId": {
            "type": "string"
          }
        }
      },
      "actor": {
        "properties": {
          "callerType": {
            "type": "string"
          },
          "email": {
            "type": "string"
          },
          "profileId": {
            "type": "long"
          },
          "key": {
            "type": "string"
          }
        }
      },
      "ownerDomain": {
        "type": "string"
      },
      "ipAddress": {
        "type": "string"
      },
      "events": {
        "properties": {
          "type": {
            "type": "string"
          },
          "name": {
            "type": "string"
          },
          "parameters": {
            "properties": {
              "name": {
                "type": "string"
              },
              "value": {
                "type": "string"
              },
              "intValue": {
                "type": "long"
              },
              "boolValue": {
                "type": "boolean"
              }
            }
          }
        }
      }
    }
  }
}'

完成此操作后，您現在可以使用批量調用為遵循上述結構的report reports#activities文檔建立索引。 批量調用的語法在此處進行了精確定義，即您需要一個命令行（要做的事情），在下一行之后是文檔源（要進行索引的工作），其中不得包含任何新行！

因此，您需要像這樣重新格式化您的documents.json文件（確保在第二行之后添加新行）。 還要注意，我添加了一些虛擬數據來說明該過程：

{"index": {"_index": "reports", "_type": "activity"}}
{"kind":"reports#activities","nextPageToken":"string","items":[{"kind":"audit#activity","id":{"time":"2016-05-31T00:00:00.000Z","uniqueQualifier":1,"applicationName":"string","customerId":"string"},"actor":{"callerType":"string","email":"string","profileId":1,"key":"string"},"ownerDomain":"string","ipAddress":"string","events":[{"type":"string","name":"string","parameters":[{"name":"string","value":"string","intValue":1,"boolValue":true}]}]}]}