Using firestore data with BigQuery

Question

Is there any automatic/manual way to export data from firestore database to a BigQuery table?

Tried to look around, it looks like there's no way to export data from firestore without using code. Any news about this one?

Thanks.

Answer 1

The simplest way to import Firestore data into BigQuery which does not require coding is using the command line. You can first export the data using the command below. Note that for importing the data into BigQuery you can only import specific collections and not all your documents in one batch.

gcloud beta firestore export gs://[BUCKET_NAME] --collection-ids='[COLLECTION_ID]'

Next, in the bucket you specified above, you will find a folder named after the timestamp of your export. Navigate the directories, locate the file ending with the file extension “export_metadata” and use its file path as the import source. You can then import the data into BigQuery using the command below:

bq --location=[LOCATION] load --source_format=DATASTORE_BACKUP [DATASET].[TABLE] [PATH_TO_SOURCE]

Answer 2

The best way to do this now is to use the official Firebase extension for exporting data in real-time from Firestore to BigQuery: https://github.com/firebase/extensions/tree/master/firestore-bigquery-export

Configuration can be done through the console or the CLI without using any code. The configured extension syncs data from Firestore to BigQuery in essentially realtime.

The extension creates an event listener on the Firestore collection you configure and a cloud function to sync data from Firestore to BigQuery.