Merge bigquery table with google cloud postgres table with federated query

Question

I am trying to merge a bigquery table (target) with a google cloud postgres table (source) with a federated query. However it looks like bigquery will not accept a federated query in the "using" clause.

Syntax error: Expected "(" or keyword SELECT or keyword WITH but got identifier "EXTERNAL_QUERY" at [3:9]

My query looks something like the below.

MERGE bigquery_dataset.bigquery_table TARGET
USING (
    EXTERNAL_QUERY("projects/company-co/locations/us/connections/company","SELECT * FROM postgres_schema.postgres_table")
        ) SOURCE

ON target.id = source.id

WHEN MATCHED THEN ...
WHEN NOT MATCHED BY TARGET THEN ...
WHEN NOT MATCHED BY SOURCE THEN ...

Are there any known workarounds for this type of functionality? Or is there any other way to perform this type of merge?

Answer 1

As per your requirement if you want to run federated queries in BigQuery where your external data source is located in Cloud PostgreSQL instance, you need to define the source dataset using the SQL function ie EXTERNAL_QUERY

The error you are getting: “Syntax error: Expected "(" or keyword SELECT or keyword WITH but got identifier "EXTERNAL_QUERY" at [3:9]” is because you are missing out the SELECT statement before your EXTERNAL_QUERY.

As per this doc , the syntax should be:

SELECT * FROM EXTERNAL_QUERY(connection_id, external_database_query[, options]);

I tried running the federated query in BigQuery where the source is in Cloud PostgreSQL and it is working as expected.

SQL QUERY:

MERGE myproject.demo.tab1 TARGET
USING (
  select * from EXTERNAL_QUERY("projects/myproject/locations/us-central1/connections/sqltobig", "SELECT * FROM entries;")
       ) SOURCE

ON target.entryID = source.entryID

WHEN MATCHED THEN
DELETE

WHEN NOT MATCHED THEN
INSERT(guestName, content, entryID)
VALUES(guestName, content, entryID)