從bash腳本創建JSON后將JSON響應轉換為CSV

Question

我有一個bash腳本，該腳本使用curl命令在網站上檢索一些JSON信息。

我正在嘗試從這些JSON結果中選擇一些特定信息，並將其填充到CSV文件中。 下面是我所得到的。

首先，這里是我用來從網站提取JSON數據的bash腳本。 （出於安全原因，我已刪除會話ID並更改了信息）

#!/bin/bash
#this script takes in the id of any customer and gives back the customer details in json format

mdid=("abcdefg" "hijklmno")
result=$(
for i in "${mdid[@]}"
do
#curl command to connect to the session and use $i as the variable for the mdid above.
curl "https://www.test.com/api/customer/$i" -H "Cookie: xxx";  -H "X-Requested-With: XMLHttpRequest" -w "\n" -H "Connection: keep-alive" --compressed
done;
)
#printing to test.txt in same folder as bash script
echo "$result" > test.json

結果是將以下條目輸入到test.json中，兩行分開

{"accountMode":"Live","acquirer":"TEST","acquirerConstraints":{"cardTypes":["MASTERCARD","MAESTRO","VISA"],"cvcRegexp":"^[0-9]{3}$","cvcRequired":true,"maxAmount":500000,"minAmount":50},"acquirerDetails":{"TEST":"Studio","ERROR_LIST":[],"MERCHANT_CODE":"218331","VALID":true,"_mId":"T712484","_status":"INPROCESS","email":"test7@gmail.com","name":"Studio","valid":true},"acquirerValidations":null,"allowedCurrencies":["EUR","USD","GBP"],"apiKeyPairs":[{"accountMode":"Live","label":"Virtual Terminal","publishableKey":"niunibiubniunijknkjknj","source":"VIRTUAL_TERMINAL"},{"accountMode":"Live","label":"Default","publishableKey":"iiuhiuhiu","source":"ECOMMERCE"}],"appLogoUrl":null,"applicationId":"541d75e0-7db8b343a31f","authorizationCode":"","closedDate":null,"closureReason":null,"declineAvsAddressFailure":false,"declineAvsZipFailure":false,"declineCvcFailure":false,"defaultCurrency":"EUR","descriptor":null,"email":"test1@gmail.com","id":"ddddeff","invitationCode":null,"locale":"en_IE","merchantApplication":{"accountNumber":null,"acquirer":"TEST","annualAmount":null,"annualVolume":null,"applicationType":"APPROVAL","bankName":"UNKNOWN","brand":null,"businessAddress":"54 My St, 1","businessAddress2":null,"businessCity":"Abbey","businessCountry":"IRL","businessPhone":null,"businessState":"DUBLIN","businessZip":null,"data":null,"email":"test@gmail.com","escalationPhone":null,"fax":null,"legalName":"UAB \"Studio\"","maxTransactionAmount":null,"mccCode":"5712","merchantPromotionCode":null,"mobile":null,"monthlyAmount":null,"monthlyVolume":null,"ownerFirstName":"tlana","ownerLastName":"nava","phone":"37647","GuideAccepted":null,"privacyAccepted":true,"privacyVersion":"1a","referenceId":"9104d65i08d071","routingNumber":null,"singleTransactionAmount":null,"statementName":"UAB \"Studio\"","taxId":null,"termsAccepted":true,"termsVersion":"1a","url":"http://www.design.lt"},"merchantId":"12484","merchantPromotionCode":null,"mposEnabled":true,"name":"Studio","netonfiguration":null,"onboardedDate":1505513232485,"onboardingMethod":null,"onboardingStatus":"INPROCESS","partner":null,"saqCompliant":false,"saqExpires":null,"settings":[{"key":"MERCHANT_DETAILS","value":"{\"zip\":\"Wicklow\",\"phone\":\"342647\",\"email\":\"suppoor@outlook.com\",\"address\":\"Bck 6\",\"state\":\"Ireland\",\"addressLine2\":\"Unit 8, Bl Par\",\"city\":\"Wicklow\"}"},{"key":"VAT_NUMBER","value":"/evzaqen/"}],"timezone":"Europe/Dublin","tinStatus":null}
{"accountMode":"Live","acquirer":"TEST","acquirerConstraints":{"cardTypes":["MASTERCARD","MAESTRO","VISA"],"cvcRegexp":"^[0-9]{3}$","cvcRequired":true,"maxAmount":500000,"minAmount":50},"acquirerDetails":{"TEST":"test","ERROR_LIST":[],"MERCHANT_CODE":"594920","MID_ASSIGNED":true,"VALID":true,"_mId":"103558","_status":"APPROVED","acquiringMid":"1036598","descriptor":"test 8885551212","email":"test@gmail.com","gatewayMid":"SIMP337","id":"SIMP337","level4Mid":"76576576","name":"test","status":"APPROVED","transactionCurrency":"USD;EUR;GBP","valid":true,"paymentGatewayKey":"ytfytfytfyt"},"acquirerValidations":null,"allowedCurrencies":["EUR","USD","GBP"],"apiKeyPairs":[],"appLogoUrl":null,"applicationId":"949bdde5-07-d8d58f4c3d01","authorizationCode":"","closedDate":null,"closureReason":null,"declineAvsAddressFailure":false,"declineAvsZipFailure":false,"declineCvcFailure":false,"defaultCurrency":"EUR","descriptor":"test85551212","email":"test@gmail.com","id":"9f3a7d7","invitationCode":null,"locale":"en_US","merchantApplication":{"accountNumber":null,"acquirer":"TEST","annualAmount":null,"annualVolume":null,"applicationType":"APPROVAL","bankName":"UNKNOWN","brand":null,"businessAddress":"123 test","businessAddress2":null,"businessCity":"Atlanta","businessCountry":"IRL","businessPhone":null,"businessState":"CARLOW","businessZip":null,"data":null,"email":"test@gmail.com","escalationPhone":null,"fax":null,"legalName":"stest","maxTransactionAmount":null,"mccCode":"521","merchantPromotionCode":null,"mobile":null,"monthlyAmount":null,"monthlyVolume":null,"ownerFirstName":"moto","ownerLastName":"test","phone":"3141212","GuideAccepted":null,"privacyAccepted":true,"privacyVersion":"1a","referenceId":"2920","routingNumber":null,"singleTransactionAmount":null,"statementName":"test","taxId":null,"termsAccepted":true,"termsVersion":"1a","url":null},"merchantId":"1036558","merchantPromotionCode":null,"mposEnabled":true,"name":"test","netonfiguration":null,"onboardedDate":1456846054925,"onboardingMethod":null,"onboardingStatus":"CLOSED","partner":null,"saqCompliant":false,"saqExpires":null,"settings":[],"timezone":"Europe/Dublin","tinStatus":"InCompliance"}

然后，我使用一個在另一個問題（ 將JSON轉換為.csv ）上找到的python腳本，該腳本處理與此類似的問題，並像下面這樣使用

import csv
import json

json_data = open("test.json")
data = json.load(json_data)

f = csv.writer(open("results.csv","wb+"))

for i in data:
    ma = data[i]["merchantApplication"]
    array = ma["email"]
    for j in array:
        f.writerow(j)

json_data.close()

當我嘗試在終端中運行此python腳本時，我得到以下錯誤：

追溯（最近一次通話最近）：文件“ merch_dets.py”，第5行，數據= json.load（json_data）文件“ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ json / init .py”，第290行，在加載中** kw）文件“ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/ init .py”，第338行，在加載返回_default_decoder.decode（s）文件“ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py”，行369，在解碼時引發ValueError（errmsg（“ Extra data“，s，end，len（s）））ValueError：額外數據：第2行第1列-第95行第1列（字符2887-250855）

我知道我呈現json行/文件的方式有問題，因為當我在在線格式化程序中嘗試一行時，它工作正常，但兩行都行不通。 我對JSON和cURL的使用經驗很少。

Answer 1

您的json文件不是有效的JSON，因為您有多個JSON根元素。

您可以使用在線驗證器進行檢查。 例如： https : //jsonformatter.curiousconcept.com/

將有效json寫入文件后，請重試。

例如，要使其有效：

代替：

{"accountMode":"Live","acquirer":"TEST","acquirerConstraints":{"cardTypes":["MASTERCARD","MAESTRO","VISA"],"cvcRegexp":"^[0-9]{3}$","cvcRequired":true,"maxAmount":500000,"minAmount":50},"acquirerDetails":{"TEST":"Studio","ERROR_LIST":[],"MERCHANT_CODE":"218331","VALID":true,"_mId":"T712484","_status":"INPROCESS","email":"test7@gmail.com","name":"Studio","valid":true},"acquirerValidations":null,"allowedCurrencies":["EUR","USD","GBP"],"apiKeyPairs":[{"accountMode":"Live","label":"Virtual Terminal","publishableKey":"niunibiubniunijknkjknj","source":"VIRTUAL_TERMINAL"},{"accountMode":"Live","label":"Default","publishableKey":"iiuhiuhiu","source":"ECOMMERCE"}],"appLogoUrl":null,"applicationId":"541d75e0-7db8b343a31f","authorizationCode":"","closedDate":null,"closureReason":null,"declineAvsAddressFailure":false,"declineAvsZipFailure":false,"declineCvcFailure":false,"defaultCurrency":"EUR","descriptor":null,"email":"test1@gmail.com","id":"ddddeff","invitationCode":null,"locale":"en_IE","merchantApplication":{"accountNumber":null,"acquirer":"TEST","annualAmount":null,"annualVolume":null,"applicationType":"APPROVAL","bankName":"UNKNOWN","brand":null,"businessAddress":"54 My St, 1","businessAddress2":null,"businessCity":"Abbey","businessCountry":"IRL","businessPhone":null,"businessState":"DUBLIN","businessZip":null,"data":null,"email":"test@gmail.com","escalationPhone":null,"fax":null,"legalName":"UAB \"Studio\"","maxTransactionAmount":null,"mccCode":"5712","merchantPromotionCode":null,"mobile":null,"monthlyAmount":null,"monthlyVolume":null,"ownerFirstName":"tlana","ownerLastName":"nava","phone":"37647","GuideAccepted":null,"privacyAccepted":true,"privacyVersion":"1a","referenceId":"9104d65i08d071","routingNumber":null,"singleTransactionAmount":null,"statementName":"UAB \"Studio\"","taxId":null,"termsAccepted":true,"termsVersion":"1a","url":"http://www.design.lt"},"merchantId":"12484","merchantPromotionCode":null,"mposEnabled":true,"name":"Studio","netonfiguration":null,"onboardedDate":1505513232485,"onboardingMethod":null,"onboardingStatus":"INPROCESS","partner":null,"saqCompliant":false,"saqExpires":null,"settings":[{"key":"MERCHANT_DETAILS","value":"{\"zip\":\"Wicklow\",\"phone\":\"342647\",\"email\":\"suppoor@outlook.com\",\"address\":\"Bck 6\",\"state\":\"Ireland\",\"addressLine2\":\"Unit 8, Bl Par\",\"city\":\"Wicklow\"}"},{"key":"VAT_NUMBER","value":"/evzaqen/"}],"timezone":"Europe/Dublin","tinStatus":null}
{"accountMode":"Live","acquirer":"TEST","acquirerConstraints":{"cardTypes":["MASTERCARD","MAESTRO","VISA"],"cvcRegexp":"^[0-9]{3}$","cvcRequired":true,"maxAmount":500000,"minAmount":50},"acquirerDetails":{"TEST":"test","ERROR_LIST":[],"MERCHANT_CODE":"594920","MID_ASSIGNED":true,"VALID":true,"_mId":"103558","_status":"APPROVED","acquiringMid":"1036598","descriptor":"test 8885551212","email":"test@gmail.com","gatewayMid":"SIMP337","id":"SIMP337","level4Mid":"76576576","name":"test","status":"APPROVED","transactionCurrency":"USD;EUR;GBP","valid":true,"paymentGatewayKey":"ytfytfytfyt"},"acquirerValidations":null,"allowedCurrencies":["EUR","USD","GBP"],"apiKeyPairs":[],"appLogoUrl":null,"applicationId":"949bdde5-07-d8d58f4c3d01","authorizationCode":"","closedDate":null,"closureReason":null,"declineAvsAddressFailure":false,"declineAvsZipFailure":false,"declineCvcFailure":false,"defaultCurrency":"EUR","descriptor":"test85551212","email":"test@gmail.com","id":"9f3a7d7","invitationCode":null,"locale":"en_US","merchantApplication":{"accountNumber":null,"acquirer":"TEST","annualAmount":null,"annualVolume":null,"applicationType":"APPROVAL","bankName":"UNKNOWN","brand":null,"businessAddress":"123 test","businessAddress2":null,"businessCity":"Atlanta","businessCountry":"IRL","businessPhone":null,"businessState":"CARLOW","businessZip":null,"data":null,"email":"test@gmail.com","escalationPhone":null,"fax":null,"legalName":"stest","maxTransactionAmount":null,"mccCode":"521","merchantPromotionCode":null,"mobile":null,"monthlyAmount":null,"monthlyVolume":null,"ownerFirstName":"moto","ownerLastName":"test","phone":"3141212","GuideAccepted":null,"privacyAccepted":true,"privacyVersion":"1a","referenceId":"2920","routingNumber":null,"singleTransactionAmount":null,"statementName":"test","taxId":null,"termsAccepted":true,"termsVersion":"1a","url":null},"merchantId":"1036558","merchantPromotionCode":null,"mposEnabled":true,"name":"test","netonfiguration":null,"onboardedDate":1456846054925,"onboardingMethod":null,"onboardingStatus":"CLOSED","partner":null,"saqCompliant":false,"saqExpires":null,"settings":[],"timezone":"Europe/Dublin","tinStatus":"InCompliance"}

采用：

[
 {"accountMode":"Live","acquirer":"TEST","acquirerConstraints":{"cardTypes":["MASTERCARD","MAESTRO","VISA"],"cvcRegexp":"^[0-9]{3}$","cvcRequired":true,"maxAmount":500000,"minAmount":50},"acquirerDetails":{"TEST":"Studio","ERROR_LIST":[],"MERCHANT_CODE":"218331","VALID":true,"_mId":"T712484","_status":"INPROCESS","email":"test7@gmail.com","name":"Studio","valid":true},"acquirerValidations":null,"allowedCurrencies":["EUR","USD","GBP"],"apiKeyPairs":[{"accountMode":"Live","label":"Virtual Terminal","publishableKey":"niunibiubniunijknkjknj","source":"VIRTUAL_TERMINAL"},{"accountMode":"Live","label":"Default","publishableKey":"iiuhiuhiu","source":"ECOMMERCE"}],"appLogoUrl":null,"applicationId":"541d75e0-7db8b343a31f","authorizationCode":"","closedDate":null,"closureReason":null,"declineAvsAddressFailure":false,"declineAvsZipFailure":false,"declineCvcFailure":false,"defaultCurrency":"EUR","descriptor":null,"email":"test1@gmail.com","id":"ddddeff","invitationCode":null,"locale":"en_IE","merchantApplication":{"accountNumber":null,"acquirer":"TEST","annualAmount":null,"annualVolume":null,"applicationType":"APPROVAL","bankName":"UNKNOWN","brand":null,"businessAddress":"54 My St, 1","businessAddress2":null,"businessCity":"Abbey","businessCountry":"IRL","businessPhone":null,"businessState":"DUBLIN","businessZip":null,"data":null,"email":"test@gmail.com","escalationPhone":null,"fax":null,"legalName":"UAB \"Studio\"","maxTransactionAmount":null,"mccCode":"5712","merchantPromotionCode":null,"mobile":null,"monthlyAmount":null,"monthlyVolume":null,"ownerFirstName":"tlana","ownerLastName":"nava","phone":"37647","GuideAccepted":null,"privacyAccepted":true,"privacyVersion":"1a","referenceId":"9104d65i08d071","routingNumber":null,"singleTransactionAmount":null,"statementName":"UAB \"Studio\"","taxId":null,"termsAccepted":true,"termsVersion":"1a","url":"http://www.design.lt"},"merchantId":"12484","merchantPromotionCode":null,"mposEnabled":true,"name":"Studio","netonfiguration":null,"onboardedDate":1505513232485,"onboardingMethod":null,"onboardingStatus":"INPROCESS","partner":null,"saqCompliant":false,"saqExpires":null,"settings":[{"key":"MERCHANT_DETAILS","value":"{\"zip\":\"Wicklow\",\"phone\":\"342647\",\"email\":\"suppoor@outlook.com\",\"address\":\"Bck 6\",\"state\":\"Ireland\",\"addressLine2\":\"Unit 8, Bl Par\",\"city\":\"Wicklow\"}"},{"key":"VAT_NUMBER","value":"/evzaqen/"}],"timezone":"Europe/Dublin","tinStatus":null},
{"accountMode":"Live","acquirer":"TEST","acquirerConstraints":{"cardTypes":["MASTERCARD","MAESTRO","VISA"],"cvcRegexp":"^[0-9]{3}$","cvcRequired":true,"maxAmount":500000,"minAmount":50},"acquirerDetails":{"TEST":"test","ERROR_LIST":[],"MERCHANT_CODE":"594920","MID_ASSIGNED":true,"VALID":true,"_mId":"103558","_status":"APPROVED","acquiringMid":"1036598","descriptor":"test 8885551212","email":"test@gmail.com","gatewayMid":"SIMP337","id":"SIMP337","level4Mid":"76576576","name":"test","status":"APPROVED","transactionCurrency":"USD;EUR;GBP","valid":true,"paymentGatewayKey":"ytfytfytfyt"},"acquirerValidations":null,"allowedCurrencies":["EUR","USD","GBP"],"apiKeyPairs":[],"appLogoUrl":null,"applicationId":"949bdde5-07-d8d58f4c3d01","authorizationCode":"","closedDate":null,"closureReason":null,"declineAvsAddressFailure":false,"declineAvsZipFailure":false,"declineCvcFailure":false,"defaultCurrency":"EUR","descriptor":"test85551212","email":"test@gmail.com","id":"9f3a7d7","invitationCode":null,"locale":"en_US","merchantApplication":{"accountNumber":null,"acquirer":"TEST","annualAmount":null,"annualVolume":null,"applicationType":"APPROVAL","bankName":"UNKNOWN","brand":null,"businessAddress":"123 test","businessAddress2":null,"businessCity":"Atlanta","businessCountry":"IRL","businessPhone":null,"businessState":"CARLOW","businessZip":null,"data":null,"email":"test@gmail.com","escalationPhone":null,"fax":null,"legalName":"stest","maxTransactionAmount":null,"mccCode":"521","merchantPromotionCode":null,"mobile":null,"monthlyAmount":null,"monthlyVolume":null,"ownerFirstName":"moto","ownerLastName":"test","phone":"3141212","GuideAccepted":null,"privacyAccepted":true,"privacyVersion":"1a","referenceId":"2920","routingNumber":null,"singleTransactionAmount":null,"statementName":"test","taxId":null,"termsAccepted":true,"termsVersion":"1a","url":null},"merchantId":"1036558","merchantPromotionCode":null,"mposEnabled":true,"name":"test","netonfiguration":null,"onboardedDate":1456846054925,"onboardingMethod":null,"onboardingStatus":"CLOSED","partner":null,"saqCompliant":false,"saqExpires":null,"settings":[],"timezone":"Europe/Dublin","tinStatus":"InCompliance"}
]

Answer 2

除了Bret Weinraub提到的JSON格式更改之外 ，您還需要更改迭代JSON元素的代碼。 現在，您正在使用新數組，就好像它是字典一樣。

另外，您將字段"email"的字符串值（在給定的示例中始終只包含一個字符串）存儲在稱為array的變量中，並循環訪問此變量以輸出電子郵件地址。 這將在不同的行中輸出電子郵件地址的所有字符。

按照Bret Weinraub的建議更改您的JSON格式，並像這樣更改您的迭代（我可以自由地重命名變量，以減少混亂：

import csv
import json

json_data = open("test.json")
data = json.load(json_data)

f = csv.writer(open("results.csv","wb+"))
for entry in data:
    if "merchantApplication" in entry:
        ma = entry["merchantApplication"]
        if "email" in ma:
            f.writerow([ma["email"]])

json_data.close()

請注意， writerow還將數組作為參數，數組中的每個條目都對應於結果CSV文件的不同列。 有關詳細信息，請參見Python csv文檔。 由於您的示例代碼似乎僅創建一列，因此我將ma["email"]直接插入到writerow在此處寫入行的數組中。

entry是您的字典（在新的JSON列表中每行有一個字典）。

我在代碼示例中包含了檢查，如果它們格式錯誤/不包含所需的數據，將跳過JSON條目。 此外，如果解決方案沒有破壞JSON格式，則該解決方案足夠強大，足以處理換行符-您的JSON字典不必每個都排成一行。

更新：Unicode

如評論中所述，您遇到Unicode編碼錯誤。 這是因為您的輸入是Unicode編碼的，而您的輸出是Ascii代碼。 這個問題總是有點煩人。 Python CSV軟件包不能很好地支持Unicode，但是有一個名為unicodecsv的直接替代品 ，可以輕松解決您的問題。 您只需要將導入交換為unicodecsv。 示例代碼如下：

import unicodecsv as csv
import json

json_data = open("test.json")
data = json.load(json_data)

f = csv.writer(open("results.csv","wb+"))
for entry in data:
    if "merchantApplication" in entry:
        ma = entry["merchantApplication"]
        if "email" in ma:
            f.writerow([ma["email"]])

json_data.close()

另外，您可以手動對所有數據進行編碼-我通常會嘗試避免這種情況，而只使用Unicode字符串即可。

Answer 3

恕我直言，最簡單的方法是先按行處理文件，然后將每行（以字符串形式顯示）轉換為字典：

...
json_data = open("test.json")
for line in json_data:
    data = json.loads(line)
    ma = data["merchantApplication"]
    array = ma["email"]
    f.writerow(array)
...

當心，以上代碼將創建一個包含單個字段的csv文件...