Dataprep - Append data to BigQuery table

Question

I'm using GCP's dataprep to join several csv files with the same column structure, treat some data and then write to a BigQuery database.

I have to record this data in BigQuery. Can I include this data from the dataprep and append them in a BigQuery table?

Answer 1

Yes, there is possibility to truncate data or append data to BigQuery table. In the output step DataPrep step in BigQuery table selection you can set that will be appended to table.

Answer 2

Yes, you can include your data from DataPrep and append them in a BigQuery table.

Before running the job, in the "Run Job on DataFlow" section:

1. Click on the action, since you are using BigQuery for the output, should look like "Create-BigQuery"
2. In the next windows choose you output table
3. In the left panel select "Append to this table every run"
4. Click on update

Now, when you run the Job, this will append your data.

The following documentation can be useful.

Answer 3

My difficult is in "Connect your data" step. I have planty of tables with the prefix events_ and I want than all. Naturally My intuition is to parameterize doing that events_* but I do not have this option in bigquery table ingestion (connect your data step).