将 Collections 导出到 BigQuery

Question

I have tried to export collection to BigQuery using Export Collections to BigQuery extension.
But I am not able to export all collections.

I need to export all collection and subcollection to BiqQuery using Export Collections to BigQuery extensions

I am able to export only one collection but can't export all.

Could anyone please help me to export all collections and subcollections?

Answer 1

You can try to export the information to a bucket or a local file by running the command gcloud firestore export gs://mybucket/my/path as mentioned here . This will export all the collections with a single command.

I recommend you to export it to GCS, however, you can load the info to BigQuery by following one of these methods .

There most likely is an issue with the extension you are using since extensions are still on beta , but you can still open a Feature request so that they add more functionalities to the extension in the future and/or report anybug directly .

Hope you find this useful!

Answer 2

You can install a second extension "stream collections to biquery" to have a second collection exported to BQ. Just go to the menu extensions and you did the first time

Answer 3

One thing that was helpful for me was to go to the Cloud Functions section for my Firebase project, hop over to the GCP console side of things, and download a zip of the source code deployed for my ext-firestore-bigquery-export-fsexportbigquery function.

In there, I noticed that the value of process.env.COLLECTION_PATH ultimately gets passed through to functions.firestore.document() which is just constructing a namespaced collection path, where wildcards can be used at any level in the path...and therein lies the magic.

If you set your Collection Path to simply be a wildcard variable, such as "{AnyCollection}", then the function will capture events for any and all collections in your Firestore DB. Only one instance of the extension is needed.

Overall solution:

• Set Collection Path to "{AnyCollection}"
• Set Table ID to something indicating that it's multi collection, such as "AnyCollection"
• It can be helpful to set Wildcard Column field with Parent Firestore Document IDs to true, although not required.
  • You also could interrogate the name of the collection out of the document_name column in your BigQuery table, which includes paths such as "projects/my-firebase-project/databases/(default)/documents/my-firestore-collection/U5OeEv7jZCY8ja0g2gLr".