使用 Python 从 AWS Lambda 中的 BigQuery 客户端提取 JSON 对象

Question

I am running a SQL query via the google.cloud.bigquery.Client.query package in AWS lambda (Python 2.7 runtime). The native BQ object extracted from a query is the BigQuery Row() ie,

Row((u'exampleEmail@gmail.com', u'XXX1234XXX'), {u'email': 0, u'email_id': 1})

I need to convert this to Json, ie,

[{'email_id': 'XXX1234XXX', 'email': 'exampleEmail@gmail.com'}]

When running locally, I am able to just call the python Dict function on the row to transform it, ie,

queryJob = bigquery.Client.query(sql)
list=[]
for row in queryJob.result():
    ** at this point row = the BQ sample Row object shown above **
    tmp = dict(row)
    list.append(tmp)`

but when I load this into AWS Lambda it throws the error:

ValueError: dictionary update sequence element #0 has length 22; 2 is required

I have tried forcing it in different ways, breaking it out into sections etc but cannot get this into the JSON format desired.

I took a brief dive into the rabbit hole of transforming the QueryJob into a Pandas dataframe and then from there into a JSON object, which also works locally but runs into numpy package errors in AWS Lambda which seems to be a bit of a known issue.

I feel like this should have an easy solution but just haven't found it yet.

Answer 1

Try doing it like this

`

L = []
sql = (#sql_statement)
query_job = client.query(sql)  # API request
query_job.result()
for row in query_job:
    email_id= row.get('email_id')
    email= row.get('email')
    L.append([email_id, email])

`