仅将唯一值上传到 Big Query

Question

I am trying to load a CSV file to the Big Query. My CSV file could contain duplicate rows. I am seeking help to know if there is a way, I could upload only the unique values from the CSV file using bq load command from CLI.

For example, if my CSV file contain below values:

emp_id, emp_name
1,a
2,b
3,c
1,a
4,d
5,e
3,c

I want only the unique values to be uploaded to the Big Query Table:

emp_id, emp_name
1,a
2,b
3,c
4,d
5,e

Currently I am removing the duplicate values manually before uploading to the Big Query Table. I am expecting to know if there is a switch/parameter, I could use to upload only unique values using "bq load" command from cloud shell.

Answer 1

bq load is not capable of applying any transform to your data. What you can do is pre process your file prior to loading it via bq load . You can use the command below in cloud shell to remove duplicates:

head -1 test.csv > temp && sort -u test.csv | head -n -1 > temp_2 && cat temp temp_2 > new_test.csv && rm temp*

Assuming test.csv contains your data, the command above gets the header saving it to temp . Then separately it gets the unique values and saves it to temp_2 . Build the clean file by concatenating temp and temp_2 and naming the file as new_test.csv .

You can now use new_test.csv as input in bq load . See test below:

But if you have bigger data than this, I recommend using a programming language to create a logic to clean your data.