将 XML 转换为 BigQuery 的 JSON 可加载结构

Question

I'm learning python on the job and need help improving my solution.

I need to load XML data into BigQuery .

I have it working but not sure if I have done it in a sensible way.

I call an API that returns an XML structure. I use ElementTree to parse the XML and use tree.iter() to return the tags and text from the XML. Printing my tags and text with:

for node in tree.iter():
    print(f'{node.tag}, {node.text}')

Returns:

Tag              Text
Responses        None
Response         None
ResponseId       393
ResponseText     Please respond “Has this loaded” 
ResponseType     single
ResponseStatus   0

The Responses tag appears only once per API call but Response through to ResponseStatus are repeating groups, ResponseId is the key for each group. Each call would return less than a 100 repeating groups.

There is a key returned in the header, Response_key , that is the parent of all ResponseIds . My aim is to take this data, convert to JSON and stream to BigQuery.

The table structure I need is:

ResponseKey, ResponseID, Response, ResponseText, ResponseType , ResponseStatus

The approach I use is

1. Use tree.iter() to loop and create a list

   node_list = [] for node in tree.iter(): node_list.append(node.tag) node_list.append(node.text)

2. Use itertools to group the list (this I found a difficult step)

    r = 'Response ' response _split = [list(y) for x, y in itertools.groupby(node_list, lambda z: z == r) if not x]

which returns:

[['Responses', 'None'], ['None', 'ResponseId', '393', 'ResponseText', Please respond “Has this loaded” 
"', 'ResponseType', 'single', 'ResponseStatus', '0'], ['None', 'ResponseId', '394', 'ResponseText', Please confirm “Connection made” "', 'ResponseType', 'single', 'ResponseStatus', '0']]

3. Load into a Pandas data frame, remove any double quotes in case that causes BigQuery any issues.
4. Add ResponseKey as a column to the dataframe.
5. Convert data frame to JSON and pass to load_table_from_json .

It works but not sure if it is sensible.

Any suggested improvements would be appreciated.

Here is a sample of the XML:

{"GetResponses":"<Responses><Response><ResponseId>393938<\/ResponseId><ResponseText>Please respond to the following statement:\"The assigned task was easy to complete\"<\/ResponseText><ResponseType>single<\/ResponseType><ResponseStatus>0<\/ResponseStatus><ExtendedType>0<\/ExtendedType><\/Response><Response><ResponseId>393939<\/ResponseId><ResponseText>Did you save your  datafor later? Why\/why not?<\/ResponseText><ResponseType>text<\/ResponseType><ResponseStatus>1<\/ResponseStatus><ExtendedType>0<\/ExtendedType><\/Response><Response><ResponseId>393940<\/ResponseId><ResponseText>Did you notice how much it cost to find the item? How much was it?<\/ResponseText><ResponseType>text<\/ResponseType><ResponseStatus>0<\/ResponseStatus><ExtendedType>0<\/ExtendedType><\/Response><Response><ResponseId>393941<\/ResponseId><ResponseText>Did you select ‘signature on form’? Why\/why not?<\/ResponseText><ResponseType>text<\/ResponseType><ResponseStatus>1<\/ResponseStatus><ExtendedType>0<\/ExtendedType><\/Response><Response><ResponseId>393942<\/ResponseId><ResponseText>Was it easy to find thethe new page? Why\/why not?<\/ResponseText><ResponseType>single<\/ResponseType><ResponseStatus>1<\/ResponseStatus><ExtendedType>4<\/ExtendedType><\/Response><Response><ResponseId>393943<\/ResponseId><ResponseText>Please enter your email. So that we can track your responses, we need you to provide this for each task.<\/ResponseText><ResponseShortCode>email<\/ResponseShortCode><ResponseType>text<\/ResponseType><ResponseStatus>1<\/ResponseStatus><ExtendedType>0<\/ExtendedType><\/Response><Response><ResponseId>393944<\/ResponseId><ResponseText>Why didn't you save your  datafor later?<\/ResponseText><ResponseType>text<\/ResponseType><ResponseStatus>0<\/ResponseStatus><ExtendedType>0<\/ExtendedType><\/Response><Response><ResponseId>393945<\/ResponseId><ResponseText>Why did you save your  datafor later?<\/ResponseText><ResponseType>single<\/ResponseType><ResponseStatus>0<\/ResponseStatus><ExtendedType>4<\/ExtendedType><\/Response><Response><ResponseId>393946<\/ResponseId><ResponseText>Did you save your  datafor later?<\/ResponseText><ResponseType>single<\/ResponseType><ResponseStatus>0<\/ResponseStatus><ExtendedType>0<\/ExtendedType><\/Response><Response><ResponseId>393947<\/ResponseId><ResponseText>Why didn't you select 'signature on form'?<\/ResponseText><ResponseType>text<\/ResponseType><ResponseStatus>0<\/ResponseStatus><ExtendedType>0<\/ExtendedType><\/Response><Response><ResponseId>393948<\/ResponseId><ResponseText>Why did you select 'signature on form'?<\/ResponseText><ResponseType>text<\/ResponseType><ResponseStatus>0<\/ResponseStatus><ExtendedType>0<\/ExtendedType><\/Response><Response><ResponseId>4444449<\/ResponseId><ResponseText>Did you select ‘signature on form’?<\/ResponseText><ResponseType>single<\/ResponseType><ResponseStatus>0<\/ResponseStatus><ExtendedType>0<\/ExtendedType><\/Response><Response><ResponseId>393950<\/ResponseId><ResponseText>Why wasn't it easy to find thethe new page?<\/ResponseText><ResponseType>single<\/ResponseType><ResponseStatus>0<\/ResponseStatus><ExtendedType>4<\/ExtendedType><\/Response><Response><ResponseId>393951<\/ResponseId><ResponseText>Was it easy to find thethe new page?<\/ResponseText><ResponseType>single<\/ResponseType><ResponseStatus>0<\/ResponseStatus><ExtendedType>0<\/ExtendedType><\/Response><Response><ResponseId>393952<\/ResponseId><ResponseText>Please enter your email addressSo that we can track your responses, we need you to provide this for each task<\/ResponseText><ResponseShortCode>email<\/ResponseShortCode><ResponseType>single<\/ResponseType><ResponseStatus>0<\/ResponseStatus><ExtendedType>4<\/ExtendedType><\/Response><\/Responses>"}

A sample JSON without all the processing steps:

node_list = []
for node in tree.iter():
    node_list.append(node.tag)
    node_list.append(node.text)

json_format = json.dumps(node_list )
print(json_format)


["Responses", null, "Response", null, "ResponseId", "393938", "ResponseText", Please respond to the following statement:\"The assigned task was easy to complete"", "ResponseType", "single", "ResponseStatus", "0", "ExtendedType", "0"]

Answer 1

I'm not sure what is the required outpot, this is one way of doing it

import xml.etree.ElementTree as ET
import json

p = r"d:\tmp.xml"
tree = ET.parse(p)

root = tree.getroot()

json_dict = {}

json_dict[root.tag] = root.text

json_dict['response_list'] = []


for node in root:
    tmp_dict = {}
    for response_info in node:
        tmp_dict[response_info.tag] = response_info.text
    json_dict['response_list'].append(tmp_dict)

with open(r'd:\out.json', 'w') as of:
    json.dump(json_dict, of)