Python GAE - Call to Big Query API throws Deadline exceeded error

Question

I've been using the Big Query api and 90% of my requests failed due to this error:

HTTPException: Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/bigquery/v2/projects/poorproject/queries?alt=json

I'm performing UPDATE operation so I'm using the standard SQL instead of legacy SQL. I know that standard SQL is much slower and for this reason the Deadline error is being thrown.

I'm using GAE Endpoints API and I've tried to increase the urlfetch time but it didn't work.

Is there a solution for this?

Thanks :)

Answer 1

Without knowing more details about your scenario (eg which specific API you are trying to call), I can only speculate as to the possible causes. I suspect you are using the synchronous 'Jobs.Query' API instead of asynchronous 'Jobs.Insert' API. If so, I'd recommend using the asynchronous API since that'd help you avoid any short deadlines on your side since inserting a job is pretty fast. If you are still experiencing timeouts, filing a bug on the BigQuery issue tracker with more details is an option.