Download Big Query Table as JSON

Question

I would like to download an existing Big Query table as JSON for manipulating the one of column which has long string.

Big Query table has been ingested with Datastore backup file from App Engine to GCS. I used Big Query to read Datastore backup file from GCS and created a table out of it, which has resulted a repeated string column as very long string.

I couldn't parse the long string, so that I would need to download the table as JSON and re up to Big Query as new table. I would need an advice for this approach

Answer 1

There are 3 ways to export your data:

1. Single URI (1 file, limit 1GB, most probably you are using this)

['gs://my-bucket/file-name.json']

Creates:

gs://my-bucket/file-name.json

2. Single wildcard URI (multiple files are created each 1GB)

['gs://my-bucket/file-name-*.json']

Creates:

gs://my-bucket/file-name-000000000000.json
gs://my-bucket/file-name-000000000001.json
gs://my-bucket/file-name-000000000002.json ...

3. Multiple wildcard URIs (this needs Hadoop)

gs://my-bucket/file-name-{worker number}-*.json

Creates:

This example assumes that BigQuery creates 80 sharded files in each partition.

gs://my-bucket/file-name-1-000000000000.json
gs://my-bucket/file-name-1-000000000001.json
...
gs://my-bucket/file-name-1-000000000080.json
gs://my-bucket/file-name-2-000000000000.json
gs://my-bucket/file-name-2-000000000001.json
...
gs://my-bucket/file-name-2-000000000080.json
gs://my-bucket/file-name-3-000000000000.json
gs://my-bucket/file-name-3-000000000001.json
...
gs://my-bucket/file-name-3-000000000080.json

Read more at:

https://cloud.google.com/bigquery/exporting-data-from-bigquery