[英]What is the appropriate way to load data from Firestore into BigQuery?
I was looking at a few ways to export data out from Firestore without using export (expensive operation in the long term as it doesn't support incremental backups) to use in BigQuery and Data Studio.我正在研究几种从 Firestore 导出数据的方法,而无需在 BigQuery 和 Data Studio 中使用导出(从长远来看,这是一项昂贵的操作,因为它不支持增量备份)。
1) Using Google Pub/Sub. 1) 使用谷歌发布/订阅。 This will probably require function to both write to pub/sub and then another to trigger to BQ.
这可能需要一个函数来写入 pub/sub,然后另一个触发 BQ。
2) Using Cloud Functions to trigger from an onCreate event to write directly to a BigQuery dataset and table. 2) 使用 Cloud Functions 从 onCreate 事件触发直接写入 BigQuery 数据集和表。 (This is using table.insert)
(这是使用 table.insert)
What would the advantage be to use Pub/Sub - other than it would appear that it will cost more in the long term?使用 Pub/Sub 的好处是什么 - 除了看起来从长远来看它会花费更多?
Or is there another way I am unaware of to do this?或者还有其他我不知道的方法吗?
I'm new at this.我是新手。 Some advise and pro and cons of the above scenarios are much appreciated.
非常感谢上述情况的一些建议和利弊。
The official solution is here .官方解决方案在这里。
In case of using Cloud Functions to trigger from an onCreate event, what will you create?如果使用 Cloud Functions 从 onCreate 事件触发,您将创建什么? Create File on Cloud Storage or create Firestore Document?
在 Cloud Storage 上创建文件还是创建 Firestore 文档?
I think that in case of using Cloud Functions you should use PubSub trigger.我认为在使用 Cloud Functions 的情况下,您应该使用 PubSub 触发器。
I recommend asynchronous architecture like Pub/Sub.我推荐像 Pub/Sub 这样的异步架构。 Because rerun is easy and the scope of influence is limited.
因为重播容易,影响范围有限。
I developed sample is here .我开发的样本在这里。 I'm using Cloud Scheduler not
cron.yaml
.我使用的是 Cloud Scheduler 而不是
cron.yaml
。 The cost of Cloud Scheduler is here . Cloud Scheduler 的成本在这里。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.