嘗試使用 python 更正格式不正確的 JSON 字符串

Question

我正在嘗試使用Python “ re ”庫和 python 切片的任意組合來糾正 Kafka 在 HDFS 上使用 Cloudera 的 Hadoop 發行版提供給我們的格式不正確的 JSON 字符串。

不正確的json：

{"json_data":"{"table":"TEST.FUBAR","op_type":"I","op_ts":"2019-03-14 15:33:50.031848","current_ts":"2019-03-14T15:33:57.479002","pos":"1111","after":{"COL1":949494949494949494,"COL2":99,"COL3":2,"COL4":"            99999","COL5":9999999,"COL6":90,"COL7":42478,"COL8":"I","COL9":null,"COL10":"2019-03-14 15:33:49","COL11":null,"COL12":null,"COL13":null,"COL14":"x222263 ","COL15":"2019-03-14 15:33:49","COL16":"x222263 ","COL17":"2019-03-14 15:33:49","COL18":"2020-09-10 00:00:00","COL19":"A","COL20":"A","COL21":0,"COL22":null,"COL23":"2019-03-14 15:33:47","COL24":2,"COL25":2,"COL26":"R","COL27":"2019-03-14 15:33:49","COL28":"  ","COL29":"PBU67H   ","COL30":"            20000","COL31":2,"COL32":null}}"}

注意：開始標記附近的雙引號“ json_data ”： “ { 和末尾附近的雙引號“ null }} ” } 實際上是唯一需要刪除的錯誤（我已經在沒有額外引號的情況下對其進行了測試） )

有效且正確的 json：

{"json_data":{"table":"TEST.FUBAR","op_type":"I","op_ts":"2019-03-14 15:33:50.031848","current_ts":"2019-03-14T15:33:57.479002","pos":"1111","after":{"COL1":949494949494949494,"COL2":99,"COL3":2,"COL4":"            99999","COL5":9999999,"COL6":90,"COL7":42478,"COL8":"I","COL9":null,"COL10":"2019-03-14 15:33:49","COL11":null,"COL12":null,"COL13":null,"COL14":"x222263 ","COL15":"2019-03-14 15:33:49","COL16":"x222263 ","COL17":"2019-03-14 15:33:49","COL18":"2020-09-10 00:00:00","COL19":"A","COL20":"A","COL21":0,"COL22":null,"COL23":"2019-03-14 15:33:47","COL24":2,"COL25":2,"COL26":"R","COL27":"2019-03-14 15:33:49","COL28":"  ","COL29":"PBU67H   ","COL30":"            20000","COL31":2,"COL32":null}}}

我每小時需要使用 Pyspark 讀取40,000 到 60,000 條記錄，基礎設施團隊說我需要修復。

是否有一種快速而骯臟的方法使用 python 讀取所有字符串並刪除開頭和結尾附近的雙引號？

Answer 1

對於字符串提供我建議你堅持re正則表達式，如：

'(?<=:|\})(")(?=\}|\{)'

應該做的伎倆。 由於不需要的雙引號跟在結束的黑色或冒號之后，並且在開始或結束括號之前。

import re
import json

string = '{"json_data":"{"table":"TEST.FUBAR","op_type":"I","op_ts":"2019-03-14 15:33:50.031848","current_ts":"2019-03-14T15:33:57.479002","pos":"1111","after":{"COL1":949494949494949494,"COL2":99,"COL3":2,"COL4":"            99999","COL5":9999999,"COL6":90,"COL7":42478,"COL8":"I","COL9":null,"COL10":"2019-03-14 15:33:49","COL11":null,"COL12":null,"COL13":null,"COL14":"x222263 ","COL15":"2019-03-14 15:33:49","COL16":"x222263 ","COL17":"2019-03-14 15:33:49","COL18":"2020-09-10 00:00:00","COL19":"A","COL20":"A","COL21":0,"COL22":null,"COL23":"2019-03-14 15:33:47","COL24":2,"COL25":2,"COL26":"R","COL27":"2019-03-14 15:33:49","COL28":"  ","COL29":"PBU67H   ","COL30":"            20000","COL31":2,"COL32":null}"}}'

trimmed_string = re.sub('(?<=:|\})(")(?=\}|\{)', '', string)

data = json.loads(trimmed_string)

結果：

{'json_data': {'table': 'TEST.FUBAR', 'op_type': 'I', 'op_ts': '2019-03-14 15:33:50.031848','current_ts': '2019-03-14T15:33:57.479002', 'pos': '1111', 'after': {'COL1': 949494949494949494, 'COL2': 99, 'COL3': 2, 'COL4': '            99999', 'COL5': 9999999, 'COL6': 90, 'COL7':42478, 'COL8': 'I', 'COL9': None, 'COL10': '2019-03-14 15:33:49', 'COL11': None, 'COL12': None, 'COL13': None, 'COL14': 'x222263 ', 'COL15': '2019-03-14 15:33:49', 'COL16': 'x222263 ', 'COL17': '2019-03-14 15:33:49', 'COL18': '2020-09-10 00:00:00', 'COL19': 'A', 'COL20': 'A', 'COL21': 0, 'COL22': None, 'COL23': '2019-03-14 15:33:47', 'COL24': 2, 'COL25': 2, 'COL26': 'R', 'COL27': '2019-03-14 15:33:49', 'COL28': '  ', 'COL29': 'PBU67H   ', 'COL30': '20000', 'COL31': 2, 'COL32': None}}}