Big Query API via Apps script - I get an error on a table that is large but works on a smaller table. Is there a size limit via apps script?

Question

I am using Apps Script to send a query to BigQuery

I have a project id - "bigquerytest-282720", a dataset called "testDataSet" and 2 tables (smallTable and bigTable).

both tables have the same Schema. Table 1 has just 6 rows, table 2 has 600000

when I query table 1 (the small table) using the request:

var request = {
query: 'SELECT * ' +
  'FROM bigquerytest-282720.testDataSet.smallTable ' +
  'WHERE Member = "North"',
useLegacySql: false
}

it returns records successfully

but when I query table 2 (the big table)

var request = {
query: 'SELECT * ' +
  'FROM bigquerytest-282720.testDataSet.bigTable ' +
  'WHERE Member = "North"',
useLegacySql: false
}

it fails with the message

GoogleJsonResponseException: API call to bigquery.jobs.getQueryResults failed with error: Not found: Job bigquerytest-282720:job_6JTUvDCP7aWSJQT4FfGZLJFqwzqm (line 35, file "Code")

I can successfully query table 2 from within the console so it appears to be an Apps script limitation. When I query table 2 in the console it says that 74.3MB has been processed so perhaps there is a table size limitation when querying from Apps Script?

Any insights?

thanks

EDIT: I've now tried with tables of 300000, 100000 and 50000 rows. It works on 50000. so clearly there is a limit somewhere in Apps Script. Is this limit fixed or can I pay for more somewhere? I'm currently not paying for anything related to BigQuery but will it solve the issue if I do?

Answer 1

Urlfetch response size has a limit of 50MB . The same may apply to advanced Google services or other inbuilt services requesting external data.

Answer 2

Ahhhhhhh - I've solved it. I was creating the table in bigQuery from google drive. Turns out it wasn't actually creating the table in bigQuery it was simply linking to the csv file in google drive, The 50000 row table that worked. I actually uploaded from local machine so this actually DID create in bigQuery. So I've signed up to free trial and uploaded from google cloud storage and this then created the big table actually in bigQuery and I can now query successfully from apps script. There must be a limit on the size of file that can be queried as an external data source, When I tried to upload data previously using apps script to bigQuery, sourcing the file in google drive. it gave an error of 50mb limit so I guess that's the same for querying from google drive via bigQuery.