Uploading CSV from google cloud to Bigquery using python

Question

I'm trying to upload a CSV file to bigQuery via Google Cloud and facing formatting issues. I have two date columns(date and cancel) to convert to the required bigQuery DateTime format, I'm using this code for conversion.

df['date'] = pd.to_datetime(df['date']

this works fine for the "date" column but doesn't work for the "cancel" column, the "cancel column has some empty rows, are empty rows an issue?? And when I execute the code mentioned above, an additional column is automatically added(as a first column) to the CSV with random integer values. How to get rid of the formatting issues??

Answer 1

Me using the ELT approach, first load all your data to Bigquery and transform accordingly.

ie, Make all columns as string and load. Thus you will not get error. Then you can transform as you want in Bigquery.