Python - Use Python variable for BigQuery Magic query structure (not query parameter)

Question

I want to use a Python variable to help build the structure of a BigQuery Magic SQL query, however I'm unable to identify a way to do so. It is not a parameter for a WHERE clause or anything similar - it's the structure of the query itself.

This is the BigQuery Magic I want to use the Python variable day_of_week in:

%%bigquery df
SELECT DATE_DIFF(CURRENT_DATE(), opening_date, WEEK(WEDNESDAY)) AS diff_weeks #I want to change WEDNESDAY to day_of_week

I have tried doing the following:

params = {"day_of_week": day_of_week} #day_of_week could be either MONDAY, TUESDAY, WEDNESDAY etc.

%%bigquery df --params $params
SELECT DATE_DIFF(CURRENT_DATE(), opening_date, WEEK(@day_of_week)) as diff_weeks

However, BigQuery Magic seems to treat this as a value parameter, 'WEDNESDAY' instead of WEDNESDAY

Any ideas?

Answer 1

I've figured an alternative way to do what I want, which preserves the use of BigQuery Magic.

From my research, it seems BigQuery Magic doesn't allow for dynamic queries where you can use Python variables for the query structure. If you want to do that, you have to use the BigQuery Python Client library and create queries as strings, which aren't pretty and easy to debug.

Therefore, I found an alternative way to do the DATE_DIFF(end_date, start_date, WEEK(<WEEKDAY>)) in BigQuery SQL which allows me to use a Python variable to interchange <WEEKDAY> parameter dynamically.

params = {"day_of_week": day_of_week} #day_of_week could be Monday, Tuesday, Wednesday, etc.

%%bigquery df --params $params

SELECT COUNT(*) counts
FROM (SELECT DATE FROM UNNEST(GENERATE_DATE_ARRAY(start_date, end_date)) AS DATE)
WHERE FORMAT_DATE('%A', DATE) = @day_of_week #this is the Python variable
GROUP BY FORMAT_DATE('%A', DATE)

This SQL statement creates an array of dates using the GENERATE_DATE_ARRAY between the start_date and end_date provided. The FORMAT_DATE('%A', DATE) converts each of the dates into a weekday (ie Monday, Tuesday, Wednesday, etc.). And the effectively, by doing a GROUP BY FORMAT_DATE('%A', DATE) and a COUNT(*) , it counts the number of occurrences of each weekday.

Doing a WHERE FORMAT_DATE('%A', DATE) = @day_of_week then filters it down to the weekday you care about.