使用Pandas將JSON轉換為具有特定字段的CSV

Question

我目前正在嘗試使用Pandas將JSON文件轉換為CSV文件。

我現在使用的代碼能夠將JSON轉換為CSV文件。

import pandas as pd
json_data = pd.read_json("out1.json")
from pandas.io.json import json_normalize
df = json_normalize(json_data["events"])
df.to_csv("out.csv)

這是我的JSON文件：

{
  "events": [
    {
      "raw": "{\"level\": \"INFO\", \"message\": \"Disabled camera with QR scan on  by 80801234 at Area A\n\"}",
      "logtypes": [
        "json"
      ],
      "timestamp": 1537190572023,
      "unparsed": null,
      "logmsg": "{\"level\": \"INFO\", \"message\": \"Disabled camera with QR scan on  by 80801234 at Area A\n\"}",
      "id": "c77afb4c-ba7c-11e8-8000-12b233ae723a",
      "tags": [
        "INFO"
      ],
      "event": {
        "json": {
          "message": "Disabled camera with QR scan on  by 80801234 at Area A\n",
          "level": "INFO"
        },
        "http": {
          "clientHost": "116.197.237.29",
          "contentType": "text/plain; charset=UTF-8"
        }
      }
    },
    {
      "raw": "{\"level\": \"INFO\", \"message\": \"Employee number saved successfully.\"}",
      "logtypes": [
        "json"
      ],
      "timestamp": 1537190528619,
      "unparsed": null,
      "logmsg": "{\"level\": \"INFO\", \"message\": \"Employee number saved successfully.\"}",
      "id": "ad9c0175-ba7c-11e8-803d-12b233ae723a",
      "tags": [
        "INFO"
      ],
      "event": {
        "json": {
          "message": "Employee number saved successfully.",
          "level": "INFO"
        },
        "http": {
          "clientHost": "116.197.237.29",
          "contentType": "text/plain; charset=UTF-8"
        }
      }
    }
  ]
}

但我想要的只是JSON文件中的一些字段（ 時間戳 ， 級別 ， 消息 ）而不是全部。

我嘗試過各種方法：

df = json_normalize(json_data["timestamp"]) // gives a KeyError on 'timestamp'

df = json_normalize(json_data, 'timestamp', ['event', 'json', ['level', 'message']]) // TypeError: string indices must be integers

我哪里出錯了？

Answer 1

我認為json_normalize不打算在這個特定的方向上工作。 我可能是錯的但是從文檔來看，似乎規范化意味着“處理每個字典中的列表”。

假設data是

data = json.load(open('out1.json'))['events']

看看第一個條目

data[0]['timestamp']

1537190572023

json_normalize希望這是一個列表

[{'timestamp': 1537190572023}]

創建增強data2

我實際上並不推薦這種方法。
如果我們相應地創建data2 ：

data2 = [{**d, **{'timestamp': [{'timestamp': d['timestamp']}]}} for d in data]

我們可以使用json_normalize

json_normalize(
    data2, 'timestamp',
    [['event', 'json', 'level'], ['event', 'json', 'message']]
)

       timestamp event.json.level                                 event.json.message
0  1537190572023             INFO  Disabled camera with QR scan on  by 80801234 a...
1  1537190528619             INFO                Employee number saved successfully.

---

理解

我認為這樣做更簡單

pd.DataFrame([
    (d['timestamp'],
     d['event']['json']['level'],
     d['event']['json']['message'])
    for d in data
], columns=['timestamp', 'level', 'message'])

       timestamp level                                            message
0  1537190572023  INFO  Disabled camera with QR scan on  by 80801234 a...
1  1537190528619  INFO                Employee number saved successfully.

---

json_normalize

但沒有花哨的論點

json_normalize(data).pipe(
    lambda d: d[['timestamp']].join(
        d.filter(like='event.json')
    )
)

       timestamp event.json.level                                 event.json.message
0  1537190572023             INFO  Disabled camera with QR scan on  by 80801234 a...
1  1537190528619             INFO                Employee number saved successfully.