Error Bigquery/dataflow "Could not resolve table in Data Catalog"

Question

I'm having troubles with a job I've set up on dataflow.

Here is the context, I created a dataset on bigquery using the following path

bi-training-gcp:sales.sales_data

In the properties I can see that the data location is "US"

Now I want to run a job on dataflow and I enter the following command into the google shell

gcloud dataflow sql query ' SELECT country, DATE_TRUNC(ORDERDATE, MONTH), sum(sales) FROM bi-training-gcp.sales.sales_data group by 1,2 ' --job-name=dataflow-sql-sales-monthly --region=us-east1 --bigquery-dataset=sales --bigquery-table=monthly_sales

The query is accepted by the console and returns me a sort of acceptation message.

After that I go to the dataflow dashboard. I can see a new job as queued but after 5 minutes or so the job fails and I get the following error messages:

Error 2021-09-29T18:06:00.795ZInvalid/unsupported arguments for SQL job launch: Invalid table specification in Data Catalog: Could not resolve table in Data Catalog: bi-training-gcp.sales.sales_data

Error 2021-09-29T18:10:31.592036462ZError occurred in the launcher container: Template launch failed. See console logs.

My guess is that it cannot find my table. Maybe because I specified the wrong location/region, since my table is specified to be location in "US" I thought it would be on a US server (which is why I specified us-east1 as a region), but I tried all us regions with no success...

Does anybody know how I can solve this?

Thank you

Answer 1

This error occurs if the Dataflow service account doesn't have access to the Data Catalog API. To resolve this issue, enable the Data Catalog API in the Google Cloud project that you're using to write and run queries. Alternately, assign the roles/datacatalog.