I have a table containing the daily transactions with date column.
The table is in BigQuery and is partitioned by the date column.
What is the most effective way to query all month-end data from the table?
I tired the sql like below but it processed the whole table which is about 100GB
SELECT * FROM table
WHERE date = LAST_DAY(date , month)
It should process less bytes as the table is partitioned by the date? (like 300 mb if I just choose one specific end of month in the where clause)
SELECT * FROM table
WHERE date = "2022-11-30"
Any ways to get what I want with processing less data?
You can use the following query to filter on the last day of the current month and to process only the partition of the last day of month:
SELECT * FROM table
WHERE date = DATE_TRUNC(DATE_ADD(CURRENT_DATE('Europe/Paris'), INTERVAL 1 MONTH), MONTH) - 1;
The same query with a date
column instead of the current date:
SELECT * FROM table
WHERE date = DATE_TRUNC(DATE_ADD(your_date_column, INTERVAL 1 MONTH), MONTH) - 1;
You can minimize volume of data processed and cost by Calculating a list of In Scope last_date of the month
and apply filter condition over data partitioned tables.
Following example will explain you:-
Original data looks like as given below, output expected is highlighted record without scanning complete table
Code to achieve it is:-
with data as
(select '2020-11-20' as add1, 'Robert' as name Union all
select '2021-10-10' as add1, 'Smith' as name Union all
select '2023-9-9' as add1, 'Mike' as name Union all
select '2024-8-2' as add1, 'Donal' as name Union all
select '2025-7-31' as add1, 'Kim' as name ),
-- Calculing Inscope List of last_dates of the month
new_data as
(select add1, LAST_DAY(cast (add1 as date)) as last_dt
from data)
-- Applying filter condition on date fileds
select * from data a, new_data b
where cast (a.add1 as date)=last_dt
Output will be last record which is having last day of the month.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.