在python中將文本文件轉換為json

Question

我有多個文檔，總計約400 GB，我想將它們轉換為json格式，以便放置到elasticsearch進行分析。

每個文件約為200 MB。

原始文件如下所示：

IUGJHHGF@BERLIN:lhfrjy
0t7yfudf@WARSAW:qweokm246
0t7yfudf@CRACOW:Er747474
0t7yfudf@cracow:kui666666
000t7yf@Vienna:1йй2ц2й2цй2цц3у

它不僅具有英文字符。 key1始終用@分隔，其中城市用;分隔； 要么 ：

用代碼解析后：

#!/usr/bin/env python

# coding: utf8
import json


with open('2') as f:
   for line in f:
      s1 = line.find("@")
      rest = line[s1+1:]
      if rest.find(";") != -1:
         if rest.find(":") != -1:
            print "FOUND BOTH : ; "
            s2 = -0
         else:
            s2 = s1+1+rest.find(";")
      elif rest.find(":") != -1:
         s2 = s1+1+rest.find(":")
      else:
         print "FOUND NO : ; "
         s2 = -0

      key1 = line[:s1]
      city = line[s1+1:s2]
      description = line[s2+1:len(line)-1]

所有文件看起來像：

RRS12345 Cracow Sunflowers
RRD12345 Berin Data

解析之后，我想要輸出：

  {  
   "location_data":[  
      {  
         "key1":"RRS12345",
         "city":"Cracow",
         "description":"Sunflowers"
      },
      {  
         "key1":"RRD123dsd45",
         "city":"Berlin",
         "description":"Data"
      },
      {  
         "key1":"RRD123dsds45",
         "city":"Berlin",
         "description":"1йй2ц2й2цй2цц3у"
      }
   ]
}

在沒有英文字符的情況下，如何快速將其轉換為所需的json格式？

Answer 1

import json


def process_text_to_json():
    location_data = []
    with open("file.txt") as f:
        for line in f:
            line = line.split()
            location_data.append({"key1": line[0], "city": line[1], "description": line[2]})

    location_data = {"location_data": location_data}
    return json.dumps(location_data)

輸出樣本：

{“ location_data”：[{“ city”：“ Cracow”，“ key1”：“ RRS12345”，“ description”：“ Sunflowers”}，{“ city”：“ Berin”，“ key1”：“ RRD12345”，“ description“：” Data“}，{” city“：” Cracow2“，” key1“：” RRS12346“，” description“：”向日葵“}，{” city“：” Berin2“，” key1“：” RRD12346“ ，“ description”：“數據”}，{“ city”：“ Cracow3”，“ key1”：“ RRS12346”，“ description”：“向日葵”}，{“ city”：“ Berin3”，“ key1”：“ RRD12346”，“描述”：“數據”}]}

Answer 2

遍歷每行並形成字典。

例如：

d = {"location_data":[]}
with open(filename, "r") as infile:
    for line in infile:
        val = line.split()
        d["location_data"].append({"key1": val[0], "city": val[1], "description": val[2]})

print(d)