如何在 pandas 中实现大查询

Question

I have a dataframe df having all data from the table(select * from table) but how to implement below logic?

select
    max(ga_session_id) over (partition by event_timestamp,event_name) as ga_session_id,
    concat(user_pseudo_id,event_timestamp,event_name,dedup_id) as join_key,
    * except(ga_session_id)
from (
    select
        user_pseudo_id,
        case when event_params.key = 'ga_session_id' then event_params.value.int_value else null end as ga_session_id,
        event_timestamp,
        event_name,
        event_params.key,
        event_params.value.string_value,
        event_params.value.int_value,
        event_params.value.float_value,
        event_params.value.double_value,
        dedup_id
    from (
        select
            row_number() over(partition by user_pseudo_id, event_timestamp, event_name) as dedup_id,
            *
        from
            -- change this to your google analytics 4 export location in bigquery
            `ga4bigquery.analytics_250794857.events_*`
        where
            -- define static and/or dynamic start and end date
            _table_suffix between '20201201' and format_date('%Y%m%d',date_sub(current_date(), interval 1 day))),
        unnest(event_params) as event_params)

Answer 1

Understanding your question as "I have all my BQ table's raw data in a dataframe - how do I replicate the logic of this SQL statement in Python?":

The short answer is: this isn't a good idea. BigQuery will be a lot more efficient at processing this data than Python will (we're talking seconds vs hours as soon as you pass the ~10 million row mark).

Without knowing more about your use case, what would probably be best is to use the Python BigQuery client to run your SQL and import the results directly into a dataframe.

Your code would look something like:

from google.cloud import bigquery

client = bigquery.Client()
QUERY = (
    '... Your query '
    )
df = client.query(QUERY).to_dataframe()

See https://pypi.org/project/google-cloud-bigquery/ for more details on how to authenticate, documentation, etc.