Google Cloud Dataflow (Python): function to join multiple files

Question

I am new to Google cloud and know python to write few scripts, currently learning cloud functions and BiqQuery.

my question: I need to join a large CSV file with multiple lookup files and replace values from lookup files.

learnt that dataflow can be used to do ETL,but don't know how to write the code in Python.

can you please share your insights. Appreciate your help.

Answer 1

Rather than joining data in python, I suggest you separately extract and load the CSV and lookup data. Then run a BigQuery query that joins the data and writes the result to a permanent table . You can then delete the separately import data.