将 BigQuery 数据传输到 SFTP 的最佳方式是什么？

Question

The data is a few hundred Mb up to a few Gb. It could be running some BQ procedures and in the end a select. The values of this need to be transferred as a valid CSV to an SFTP.

Cloud functions could be problematic because of the 9 minute timeout limit and the 2Gb RAM limit.

Is there a serverless solution or do I have to run manual instances?

Answer 1

There are two scenarios I would consider:

1. Export table with standard BQ options ( here ) into GCP bucket storage . Then you can pick it up and upload it to SFTP by Cloud Run . There are containers, which are built for this, eg this one
2. Run a pipelining project. Considering you want to use simple export, I would suggest Dataflow . You can write a small Python or Java code to pick up a file and upload it to SFTP. If you would like more complex logic in processing - have a look at Dataproc