[英]Parse JSON output from API file into CSV
I am currently trying to convert a JSON output from an API request to a CSV format so i can store the results into our database. 我目前正在尝试将JSON输出从API请求转换为CSV格式,以便将结果存储到我们的数据库中。 Here is my current code for reference: 这是我当前的代码供参考:
import pyodbc
import csv
#import urllib2
import json
import collections
import requests
#import pprint
#import functools
print ("Connecting via ODBC")
conn = pyodbc.connect('DSN=DSN', autocommit=True)
print ("Connected!\n")
cur = conn.cursor()
sql = """SELECT DATA"""
cur.execute(sql)
#df = pandas.read_sql_query(sql, conn)
#df.to_csv('TEST.csv')
#print('CSV sheet is ready to go!')
rows = cur.fetchall()
obs_list = []
for row in rows:
d = collections.OrderedDict()
d['addressee'] = row.NAME
d['street'] = row.ADDRESS
d['city'] = row.CITY
d['state'] = row.STATE
d['zipcode'] = row.ZIP
obs_list.append(d)
obs_file = 'TEST.json'
with open(obs_file, 'w') as file:
json.dump(obs_list, file)
print('Run through API')
url = 'https://api.smartystreets.com/street-address?'
headers = {'content-type': 'application/json'}
with open('test1.json', 'r') as run:
dict_run = run.readlines()
dict_ready = (''.join(dict_run))
r = requests.post(url , data=dict_ready, headers=headers)
ss_output = r.text
output = 'output.json'
with open(output,'w') as of:
json.dump(ss_output, of)
print('I think it works')
f = open('output.json')
data = json.load(f)
data_1 = data['analysis']
data_2 = data['metadata']
data_3 = data['components']
entity_data = open('TEST.csv','w')
csvwriter = csv.writer(entity_data)
count = 0
count2 = 0
count3 = 0
for ent in data_1:
if count == 0:
header = ent.keys()
csvwriter.writerow(header)
count += 1
csvwriter.writerow(ent.values())
for ent_2 in data_2:
if count2 == 0:
header2 = ent_2.keys()
csvwriter.writerow(header2)
count2 += 1
csvwriter.writerow(ent_2.values())
for ent_3 in data_3:
if count3 == 0:
header3 = ent_3.keys()
csvwriter.writerow(header3)
count3 += 1
csvwriter.writerow(ent_3.values())
entity_data.close()
Sample output from API: API的示例输出:
[
{
"input_index": 0,
"candidate_index": 0,
"delivery_line_1": "1 Santa Claus Ln",
"last_line": "North Pole AK 99705-9901",
"delivery_point_barcode": "997059901010",
"components": {
"primary_number": "1",
"street_name": "Santa Claus",
"street_suffix": "Ln",
"city_name": "North Pole",
"state_abbreviation": "AK",
"zipcode": "99705",
"plus4_code": "9901",
"delivery_point": "01",
"delivery_point_check_digit": "0"
},
"metadata": {
"record_type": "S",
"zip_type": "Standard",
"county_fips": "02090",
"county_name": "Fairbanks North Star",
"carrier_route": "C004",
"congressional_district": "AL",
"rdi": "Commercial",
"elot_sequence": "0001",
"elot_sort": "A",
"latitude": 64.75233,
"longitude": -147.35297,
"precision": "Zip8",
"time_zone": "Alaska",
"utc_offset": -9,
"dst": true
},
"analysis": {
"dpv_match_code": "Y",
"dpv_footnotes": "AABB",
"dpv_cmra": "N",
"dpv_vacant": "N",
"active": "Y",
"footnotes": "L#"
}
},
{
"input_index": 1,
"candidate_index": 0,
"delivery_line_1": "Loop land 1",
"last_line": "North Pole AK 99705-9901",
"delivery_point_barcode": "997059901010",
"components": {
"primary_number": "1",
"street_name": "Lala land",
"street_suffix": "Ln",
"city_name": "North Pole",
"state_abbreviation": "AK",
"zipcode": "99705",
"plus4_code": "9901",
"delivery_point": "01",
"delivery_point_check_digit": "0"
},
"metadata": {
"record_type": "S",
"zip_type": "Standard",
"county_fips": "02090",
"county_name": "Fairbanks North Star",
"carrier_route": "C004",
"congressional_district": "AL",
"rdi": "Commercial",
"elot_sequence": "0001",
"elot_sort": "A",
"latitude": 64.75233,
"longitude": -147.35297,
"precision": "Zip8",
"time_zone": "Alaska",
"utc_offset": -9,
"dst": true
},
"analysis": {
"dpv_match_code": "Y",
"dpv_footnotes": "AABB",
"dpv_cmra": "N",
"dpv_vacant": "N",
"active": "Y",
"footnotes": "L#"
}
]
After storing the API output the trouble is trying to parse the returned output (Sample output) into a CSV format. 在存储API输出之后,问题就在于尝试将返回的输出(样本输出)解析为CSV格式。 The code im using to try to do this: 我用来尝试执行此操作的代码:
f = open('output.json')
data = json.load(f)
data_1 = data['analysis']
data_2 = data['metadata']
data_3 = data['components']
entity_data = open('TEST.csv','w')
csvwriter = csv.writer(entity_data)
count = 0
count2 = 0
count3 = 0
for ent in data_1:
if count == 0:
header = ent.keys()
csvwriter.writerow(header)
count += 1
csvwriter.writerow(ent.values())
for ent_2 in data_2:
if count2 == 0:
header2 = ent_2.keys()
csvwriter.writerow(header2)
count2 += 1
csvwriter.writerow(ent_2.values())
for ent_3 in data_3:
if count3 == 0:
header3 = ent_3.keys()
csvwriter.writerow(header3)
count3 += 1
csvwriter.writerow(ent_3.values())
entity_data.close()
returns the following error: TypeError: string indices must be integers. 返回以下错误:TypeError:字符串索引必须为整数。 And as someone kindly commented and pointed out it appears i am iterating over keys instead of the different dictionaries, and this is where I get stuck because im not sure what to do? 正如某位友善的评论并指出的那样,看来我是在遍历键而不是不同的字典,而这正是我受困的原因,因为我不确定该怎么做? From my understanding it looks like the JSON is split into 3 different arrays with JSON object for each, but that does not appear to be the case according to the structure? 根据我的理解,似乎JSON分为3个不同的数组,每个数组都有JSON对象,但是根据结构,情况似乎并非如此吗? I apologize for the length of the code, but I want some resemblance of context to what i am trying to accomplish. 对于代码的长度,我深表歉意,但我希望上下文与我要完成的工作类似。
Consider pandas's json_normalize()
method to flatten nested items into tabular df structure: 考虑熊猫的json_normalize()
方法将嵌套的项目展平为表格df结构:
import pandas as pd
from pandas.io.json import json_normalize
import json
with open('Output.json') as f:
data = json.load(f)
df = json_normalize(data)
df.to_csv('Output.csv')
Do note the components , metadata , and analysis become period-separated prefixes to corresponding values. 请注意, components , metadata和analysis变成了句点分隔的对应值的前缀。 If not needed, consider renaming columns. 如果不需要,请考虑重命名列。
You are saving request's result.text
with json. 您正在使用json保存请求的result.text
。 result.text
is a string so upon rereading it through json you get the same one long string instead of a list
. result.text
是一个字符串,因此在通过json重新读取它时,会得到一个相同的长字符串,而不是list
。 Try to write result.text
as is: 尝试按原样写入result.text
:
output = 'output.json'
with open(output,'w') as of:
of.write(ss_output)
That's the cause of TypeError:string indices must be integers
you mention. 这就是TypeError:string indices must be integers
您提到的TypeError:string indices must be integers
的原因。 The rest of your code has multiple issues. 您的其余代码有多个问题。
The data in json is a list of dicts so to get ,say , data_1
you need list comprehension like this: data_1 = [x['analysis'] for x in data]
json中的数据是一列字典,因此要获得data_1
您需要像这样的列表理解: data_1 = [x['analysis'] for x in data]
You write three types of rows into the same csv file: components, metadata and analyzis. 您将三种类型的行写入同一个csv文件中:组件,元数据和分析。 That's really odd. 真的很奇怪
Probably you have to rewrite the second half of the code: open three csv_writers one per data type, then iterate over data
items and write their fields into corresponding csv_writer. 可能您必须重写代码的后半部分:每种数据类型打开三个csv_writers,然后遍历data
项并将其字段写入相应的csv_writer。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.