Cloud Storage to BigQuery via Python

Question

I am receiving a data drop into my GCS bucket daily and have a cloud function that moves said csv data to a BigQuery Table (see code below).

    import datetime

def load_table_uri_csv(table_id):

    # [START bigquery_load_table_gcs_csv]
    from google.cloud import bigquery

    # Construct a BigQuery client object.
    client = bigquery.Client()

    # TODO(developer): Set table_id to the ID of the table to create.
    table_id = "dataSet.dataTable"

    job_config = bigquery.LoadJobConfig(
        write_disposition=bigquery.WriteDisposition.WRITE_APPEND,
        source_format=bigquery.SourceFormat.CSV, skip_leading_rows=1, autodetect=True,
    )

    uri = "gs://client-data/team/looker-client-" + str(datetime.date.today()) + ".csv"

    load_job = client.load_table_from_uri(
        uri, table_id, job_config=job_config
    )  # Make an API request.

    load_job.result()  # Waits for the job to complete.

    destination_table = client.get_table(table_id)  # Make an API request.
    print("Loaded {} rows.".format(destination_table.num_rows))
    # [END bigquery_load_table_gcs_csv]

However, the data comes with a 2 day look back resulting in repeated data in the BigQuery table.

Is there a way for me to update this cloud function to only pull in the most recent date from the csv once it is dropped off? This way I can easily avoid duplicative data within the reporting.

Or, maybe theres a way for me to run a scheduled query via BigQuery to resolve this?

For reference, the date column within the CSV comes in a TIMESTAMP schema.

Any and all help is appreciated!

Answer 1

There is seems to be no way to do this directly from Google Cloud Platform, unfortunately. You will need filter your information somehow before loading it.

You could review the information from the CSV in your code or through another medium.

It's also possible to submit a feature request for Google to consider this functionality.