如何将第 4 列的制表符分隔文本文件解析为 json 并删除某些键？

Question

我有一个 26 Gb 的文本文件，行格式如下

/type/edition /books/OL10000135M 4 2010-04-24T17:54:01.503315 {"publishers": ["Bernan Press"], "physical_format": "Hardcover", "subtitle": "9th November - 3rd December, 1992", "key": "/books/OL10000135M", "title": "Parliamentary Debates, House of Lords, Bound Volumes, 1992-93", "identifiers": {"goodreads": ["6850240"]}, "isbn_13": ["9780107805401"], "languages": [{"key": "/languages/eng"}], "number_of_pages": 64, "isbn_10": ["0107805405"], "publish_date": "December 1993", "last_modified": {"type": "/type/datetime", "value": "2010-04-24T17:54:01.503315"}, "authors": [{"key": "/authors/OL2645777A"}], "latest_revision": 4, "works": [{"key": "/works/OL7925046W"}], "type": {"key": "/type/edition"}, "subjects": ["Government - Comparative", "Politics / Current Events"], "revision": 4}

我试图只获取最后一列，即 json 和 Json 我只是想保存“标题”、“isbn 13”、“isbn 10”

我只能用这段代码保存最后一列

csv.field_size_limit(sys.maxsize)
# File names: to read in from and read out to
input_file = '../inputFile/ol_dump_editions_2019-10-31.txt'
output_file = '../outputFile/output.txt'

## ==================== ##
##  Using module 'csv'  ##
## ==================== ##
with open(input_file) as to_read:
    with open(output_file, "w") as tmp_file:
        reader = csv.reader(to_read, delimiter = "\t")
        writer = csv.writer(tmp_file)

        desired_column = [4]        # text column

        for row in reader:     # read one row at a time
            myColumn = list(row[i] for i in desired_column)   # build the output row (process)
            writer.writerow(myColumn) # write it

但这不会返回正确的 json object 而是返回旁边带有双引号的所有内容。 另外，我如何从 json 中提取某些值作为新的 json

编辑：

"{""publishers"": [""Bernan Press""], ""physical_format"": ""Hardcover"", ""subtitle"": ""9th November - 3rd December, 1992"", ""key"": ""/books/OL10000135M"", ""title"": ""Parliamentary Debates, House of Lords, Bound Volumes, 1992-93"", ""identifiers"": {""goodreads"": [""6850240""]}, ""isbn_13"": [""9780107805401""], ""languages"": [{""key"": ""/languages/eng""}], ""number_of_pages"": 64, ""isbn_10"": [""0107805405""], ""publish_date"": ""December 1993"", ""last_modified"": {""type"": ""/type/datetime"", ""value"": ""2010-04-24T17:54:01.503315""}, ""authors"": [{""key"": ""/authors/OL2645777A""}], ""latest_revision"": 4, ""works"": [{""key"": ""/works/OL7925046W""}], ""type"": {""key"": ""/type/edition""}, ""subjects"": [""Government - Comparative"", ""Politics / Current Events""], ""revision"": 4}"

编辑2：

所以我试图读取这个文件，它是一个带有以下列的制表符分隔文件：

type - 记录的类型（/type/edition、/type/work 等） key - 记录的唯一键。 (/books/OL1M 等) revision - 记录的修订号 last_modified - 最后修改的时间戳 JSON - JSON 格式的完整记录

我试图读取 JSON 文件并从该 Json 中读取“标题”、“isbn 13”、“isbn 10”作为 Z466DEEC76ECDF5FCA6D38571F6324D 文件并将其保存到 45

所以每一行都应该看起来像原来的，但只有那些键和值

Answer 1

因此，鉴于您当前的代码返回以下内容：

result = '{""publishers"": [""Bernan Press""], ""physical_format"": ""Hardcover"", ""subtitle"": ""9th November - 3rd December, 1992"", ""key"": ""/books/OL10000135M"", ""title"": ""Parliamentary Debates, House of Lords, Bound Volumes, 1992-93"", ""identifiers"": {""goodreads"": [""6850240""]}, ""isbn_13"": [""9780107805401""], ""languages"": [{""key"": ""/languages/eng""}], ""number_of_pages"": 64, ""isbn_10"": [""0107805405""], ""publish_date"": ""December 1993"", ""last_modified"": {""type"": ""/type/datetime"", ""value"": ""2010-04-24T17:54:01.503315""}, ""authors"": [{""key"": ""/authors/OL2645777A""}], ""latest_revision"": 4, ""works"": [{""key"": ""/works/OL7925046W""}], ""type"": {""key"": ""/type/edition""}, ""subjects"": [""Government - Comparative"", ""Politics / Current Events""], ""revision"": 4}'

看起来你需要做的是：首先 - 用常规双引号替换那些双双引号，否则事情是不可解析的：

res = result.replace('""','"')

现在res可以转换为 JSON object：

import json
my_json = json.loads(res)

my_json现在看起来像这样：

{'authors': [{'key': '/authors/OL2645777A'}],
 'identifiers': {'goodreads': ['6850240']},
 'isbn_10': ['0107805405'],
 'isbn_13': ['9780107805401'],
 'key': '/books/OL10000135M',
 'languages': [{'key': '/languages/eng'}],
 'last_modified': {'type': '/type/datetime',
  'value': '2010-04-24T17:54:01.503315'},
 'latest_revision': 4,
 'number_of_pages': 64,
 'physical_format': 'Hardcover',
 'publish_date': 'December 1993',
 'publishers': ['Bernan Press'],
 'revision': 4,
 'subjects': ['Government - Comparative', 'Politics / Current Events'],
 'subtitle': '9th November - 3rd December, 1992',
 'title': 'Parliamentary Debates, House of Lords, Bound Volumes, 1992-93',
 'type': {'key': '/type/edition'},
 'works': [{'key': '/works/OL7925046W'}]}

您可以方便地从此 object 中获取所需的任何字段：

my_json['title']
# 'Parliamentary Debates, House of Lords, Bound Volumes, 1992-93'
my_json['isbn_10'][0]
# '0107805405'

Answer 2

这是一种直接的方法。 您需要重复此操作，并在读取文件的每一行时逐行提取所需的数据（在 Python 中处理文本文件读取的默认方式）。

import json

line = '/type/edition   /books/OL10000135M  4   2010-04-24T17:54:01.503315  {"publishers": ["Bernan Press"], "physical_format": "Hardcover", "subtitle": "9th November - 3rd December, 1992", "key": "/books/OL10000135M", "title": "Parliamentary Debates, House of Lords, Bound Volumes, 1992-93", "identifiers": {"goodreads": ["6850240"]}, "isbn_13": ["9780107805401"], "languages": [{"key": "/languages/eng"}], "number_of_pages": 64, "isbn_10": ["0107805405"], "publish_date": "December 1993", "last_modified": {"type": "/type/datetime", "value": "2010-04-24T17:54:01.503315"}, "authors": [{"key": "/authors/OL2645777A"}], "latest_revision": 4, "works": [{"key": "/works/OL7925046W"}], "type": {"key": "/type/edition"}, "subjects": ["Government - Comparative", "Politics / Current Events"], "revision": 4}'

csv_cols = line.split('\t')
json_data = json.loads(csv_cols[4])
#print(json.dumps(json_data, indent=4))

desired = {key: json_data[key] for key in ("title", "isbn_13", "isbn_10")}
result = json.dumps(desired, indent=4)
print(result)

Output 来自样品线：

{
    "title": "Parliamentary Debates, House of Lords, Bound Volumes, 1992-93",
    "isbn_13": [
        "9780107805401"
    ],
    "isbn_10": [
        "0107805405"
    ]
}

Answer 3

特别是因为您的示例太大，我建议使用专用库，例如 pandas ，它具有read_csv方法，甚至dask ，它支持内存不足操作。

这两个系统都会自动为您解析出报价，而 dask 会直接从磁盘“分段”执行此操作，因此您无需尝试将 26GB 加载到 RAM 中。

在这两个库中，您可以像这样访问所需的列：

data = read_csv(PATH)
data["ColumnName"]

然后，您可以使用json.loads() ( import json ) 解析这些行，也可以使用 pandas/dask json 实现。 如果您可以提供更多关于您期望的细节，我可以帮助您起草更具体的代码示例。

祝你好运！

Answer 4

我将您的数据保存到一个文件中，看看我是否可以只读取行，让我知道这是否有效：

lines = zzread.split('\n')  
temp=[] 
for to_read in lines: 
    if len(to_read) == 0:  
        break  
    new_to_read = '{' + to_read.split('{',1)[1] 
    temp.append(json.loads(new_to_read)) 
for row in temp: 
      print(row['isbn_13'])

如果可行，这应该为您创建一个 json：

lines = zzread.split('\n')  
temp=[] 
for to_read in lines: 
    if len(to_read) == 0:  
        break  
    new_to_read = '{' + to_read.split('{',1)[1] 
    temp.append(json.loads(new_to_read)) 
new_json=[]
for row in temp: 
    new_json.append({'title': row['title'], 'isbn_13': row['isbn_13'], 'isbn_10': row['isbn_10']})