Transferring data from VM Instances to BigQuery in GCP

Question

I am trying to transfer some files to BigQuery which are stored in my VM Instances. Normally we do a two steps process:

1. Transfer files from VM instances to Cloud Storage bucket.
2. Getting data from Cloud Storage bucket to BigQuery.

Now, I want to take files directly from VM Instances to BigQuery platform. Is there any way to do it?

Answer 1

You can load data directly from a readable data source (such as your local machine) by using:

• The Cloud Console or the classic BigQuery web UI
• The bq command-line tool's bq load command
• The API
• The client libraries

Please, follow the official documentation to see examples of using each way.

Moreover, if you want to stay with idea of sending your files to Cloud Storage bucket, you can think about using Dataflow templates:

• Cloud Storage Text to BigQuery ( Stream )
• Cloud Storage Text to BigQuery ( Batch )

which allows you to read text files stored in Cloud Storage, transform them using a JavaScript User Defined Function (UDF) that you provide, and output the result to BigQuery.It is automated solution.

I hope you find the above pieces of information useful.

Answer 2

The solution would be to use bq command for this. The command would be like this: bq load --autodetect --source_format=CSV xy abc.csv