How can I insert variables into BigQuery query strings using App Script?

Question

I have ran into an issue that I cannot figure out or find a solution for anywhere on google.

I have a table in BigQuery with 150k+ unique rows and 29 columns.
Each row has a REQUIRED column dateX type DATE .

In my App Script, I am taking user input from a spreadsheet (most importantly the date range), creating a new spreadsheet in a specified folder, inserting data & setting formatting, and returning the download link to the user.

The 'final' feature I need is the ability to take the date range values provided by the user and insert them into my BigQuery query string so that I may write the data from the BigQuery table for those dates ONLY.

THE PROBLEM:
No matter how I construct my query string in App Script, the query does not return any rows. Just headers.
So far, I have determined that the WHERE statement in the query is the issue. I've tried the following:

var request = {
    query: `SELECT * FROM le.table.name WHERE dateX = '2020-10-5'`
};
//
// Some of the variations I've tried (among countless others w/ different formatting):
// 'SELECT * FROM le.table.name WHERE dateX = "2020-10-5"'
// "SELECT * FROM le.table.name WHERE dateX = '2020-10-5'"
// 'SELECT * FROM le.table.name WHERE dateX between "2020-10-5" and "2020-10-5"'
// 'SELECT * FROM le.table.name WHERE cast(dateX as date) between "2020-10-5" and "2020-10-5"'
// `SELECT * FROM le.table.name WHERE dateX = ${dateFrom}`
// `SELECT * FROM le.table.name WHERE dateX between ${dateFrom} and ${dateTo}`
//
// I've also tried setting the query string as a variable & passing with no success. 
// var queryString = 'SELECT * FROM le.table.name WHERE dateX = "2020-10-5"';
// var queryString = 'SELECT * FROM le.table.name WHERE dateX = ' + '2020-10-5';

ALL of these run just fine. Job executes with no errors. But it only returns the headers, no rows:(
The same query runs just fine and returns the desired rows when I run in BigQuery directly.

FURTHERMORE, if I remove everything from WHERE onward, it runs the query WITH ALL my 150k+ rows.
If I select my columns, it returns the desired columns. But won't return ALL (*).

I am beyond confused & truly lost. Perhaps I have spent so much time on this I've developed a tunnel vision & NEED a second set of eyes to spot the issue. Any help is GREATLY appreciated. Thank you in advance.

Answer 1

Not sure I can say anything more without a sample data. If possible - perhaps you could share some anonymized version of your dataset?

My guess would be a simple typo:

var request = {
    query: `SELECT * FROM le.table.name WHERE dateX = '2020-10-5'`
};

Shouldn't it be '2020-10- 0 5'?

Answer 2

I've slept on the issue and found a solution to my problem:

When running the query directly in BigQuery, it is smart enough to know that '2020-10-05' is actually a date, and so it reads it in DATE format.

When constructing the query string in App Script, the whole this is a String , therefore the '2020-10-05' is also a string. This obviously causes WHERE to return false as it is comparing dateX , a DATE , to a String .
This is solved by using cast() on the date variables:

var request = {
    query: `SELECT * FROM le.table.name WHERE dateX between ` + 
           `cast(${dateFrom} as date) and cast(${dateTo} as date);`
}  
//
//Note: dateFrom & dateTo variables need to include ' ' in them.
//  Ex: var dateFrom = "'2020-10-05'";
//

I hope this can help anyone else dealing with App Script & BigQuery!