在BigQuery中查询所有月末数据的最有效方法

Question

I have a table containing the daily transactions with date column.
The table is in BigQuery and is partitioned by the date column.

What is the most effective way to query all month-end data from the table?
I tired the sql like below but it processed the whole table which is about 100GB

SELECT * FROM table
WHERE date = LAST_DAY(date , month)

It should process less bytes as the table is partitioned by the date? (like 300 mb if I just choose one specific end of month in the where clause)

SELECT * FROM table
WHERE date = "2022-11-30"

Any ways to get what I want with processing less data?

Answer 1

You can use the following query to filter on the last day of the current month and to process only the partition of the last day of month:

SELECT * FROM table
WHERE date = DATE_TRUNC(DATE_ADD(CURRENT_DATE('Europe/Paris'), INTERVAL 1 MONTH), MONTH) - 1;

The same query with a date column instead of the current date:

SELECT * FROM table
WHERE date = DATE_TRUNC(DATE_ADD(your_date_column, INTERVAL 1 MONTH), MONTH) - 1;

Answer 2

You can minimize volume of data processed and cost by Calculating a list of In Scope last_date of the month and apply filter condition over data partitioned tables.

Following example will explain you:-

Original data looks like as given below, output expected is highlighted record without scanning complete table

Code to achieve it is:-

with data as 
(select '2020-11-20' as add1, 'Robert' as name Union all
select '2021-10-10' as add1, 'Smith' as name Union all
select '2023-9-9' as add1, 'Mike' as name Union all
select '2024-8-2' as add1, 'Donal' as name Union all
select '2025-7-31' as add1, 'Kim' as name ),

-- Calculing Inscope List of last_dates of the month
new_data as
 (select add1, LAST_DAY(cast (add1 as date)) as last_dt
 from data)

-- Applying filter condition on date fileds 
select * from data a, new_data b
where cast (a.add1 as date)=last_dt

Output will be last record which is having last day of the month.