SQL/BigQuery 如何对这些数据进行会话？

Question

I'm struggling to figure out how to write a query that will properly aggregate the below sample data. This sample data represents the output of an aggregate query that I then need to filter down further. I'm not 100% sure on this, but I think I need to sessionize this data, where the session starts on the first row of a resource_name that is null for complete and ends when that resource name has true in column complete , with a lag time of ~ 30 minutes.

I unfortunately don't have the ability to change the data, and all of the output below is from a single table with a query that aggregates the fields as such.

Example Data:

|resource_name|operation_type|initiate|complete|timestamp             |
|-------------|--------------|--------|--------|----------------------|
|foo          |full          |true    |null    |2021-11-01-5:51:46 UTC|
|foo          |full          |null    |true    |2021-11-01-5:51:49 UTC|
|foo          |incomplete    |null    |null    |2021-11-01-7:02:22 UTC|  <--- foo begins
|foo          |incomplete    |null    |null    |2021-11-01-7:02:37 UTC|
|foo          |incomplete    |null    |null    |2021-11-01-7:03:19 UTC|
|baz          |incomplete    |null    |null    |2021-11-01-7:03:25 UTC|
|baz          |incomplete    |null    |null    |2021-11-01-7:03:29 UTC|
|foo          |full          |true    |null    |2021-11-01-7:03:31 UTC|
|foo          |full          |null    |true    |2021-11-01-7:12:55 UTC|  <--- foo ends
|bar          |incomplete    |null    |null    |2021-11-01-7:39:22 UTC|  <--- bar starts
|bar          |incomplete    |null    |null    |2021-11-01-7:40:37 UTC|
|baz          |incomplete    |null    |null    |2021-11-01-7:41:37 UTC|
|baz          |incomplete    |null    |null    |2021-11-01-7:41:39 UTC|
|baz          |incomplete    |null    |null    |2021-11-01-7:41:45 UTC|
|bar          |incomplete    |null    |null    |2021-11-01-7:44:19 UTC|
|bar          |incomplete    |null    |null    |2021-11-01-7:44:58 UTC|
|bar          |full          |true    |null    |2021-11-01-7:45:31 UTC|
|bar          |full          |null    |true    |2021-11-01-7:47:55 UTC|  <--- bar ends
|bar          |incomplete    |null    |null    |2021-11-01-9:38:22 UTC|  <--- bar starts again 
|bar          |incomplete    |null    |null    |2021-11-01-9:40:37 UTC|
|bar          |full          |true    |null    |2021-11-01-9:45:31 UTC|
|bar          |full          |null    |true    |2021-11-01-9:51:55 UTC|  <--- bar ends again

What I'm trying to do is find the timestamp differences for each resource_name between the first incomplete operation_type and the next full operation_type where complete = true , for each resource_name.

So, in this case, I would return one value for foo , and two values for bar . foo has one initial incomplete operation_type , and one full operation_type with complete = true , and bar has two instances of the same.

My results should be (duration not computed so you can see timestamps that should be picked up, sorted DESC):

|resource_name|duration                                                        |
|-------------|----------------------------------------------------------------|
|bar          | timestamp_diff(2021-11-01-9:51:55 - 2021-11-01-9:38:22, SECOND)|
|foo          | timestamp_diff(2021-11-01-7:12:55 - 2021-11-01-7:02:22, SECOND)|
|bar          | timestamp_diff(2021-11-01-7:47:55 - 2021-11-01-7:39:22, SECOND)|

Answer 1

Based on the information you have provided I have come with this solution:

I follow these steps to recreate it on my side:

1. create a .csv table based on the data you provide in your post.
2. created a dataset and a table
3. filled the table with the .csv file
4. run below query.

So to explain the though process, the approach should be about getting your success runs first then use it to go the minimum incomplete values for each run. For what I see in the way you have more than one error inside an hour so we split it based on an hour to cover that scenario.

CSV

foo,full,true,,2021-11-01 05:51:46
foo,full,,true,2021-11-01 05:51:49
foo,incomplete,,,2021-11-01 07:02:22
foo,incomplete,,,2021-11-01 07:02:37
foo,incomplete,,,2021-11-01 07:03:19
baz,incomplete,,,2021-11-01 07:03:25
baz,incomplete,,,2021-11-01 07:03:29
foo,full,true,,2021-11-01 07:03:31
foo,full,,true,2021-11-01 07:12:55
bar,incomplete,,,2021-11-01 07:39:22
bar,incomplete,,,2021-11-01 07:40:37
baz,incomplete,,,2021-11-01 07:41:37
baz,incomplete,,,2021-11-01 07:41:39
baz,incomplete,,,2021-11-01 07:41:45
bar,incomplete,,,2021-11-01 07:44:19
bar,incomplete,,,2021-11-01 07:44:58
bar,full,true,,2021-11-01 07:45:31
bar,full,,true,2021-11-01 07:47:55
bar,incomplete,,,2021-11-01 09:38:22
bar,incomplete,,,2021-11-01 09:40:37
bar,full,true,,2021-11-01 09:45:31
bar,full,,true,2021-11-01 09:51:55

Bigquery Code

with operations_sucess as (
    select f.resource_name,
    f.operation_type,
    f.timestamp,
    extract(hour from f.timestamp) as hour, 
    UNIX_SECONDS(f.timestamp) as unixsecs
    from <MY TEMP BIGQUERY TABLE> f 
    where f.operation_type='full' and complete = True 
    group by f.resource_name,f.operation_type,f.timestamp
    order by f.timestamp
)
select s.resource_name,
timestamp_diff(TIMESTAMP_SECONDS(MAX(ss.unixsecs)),MIN(s.timestamp),SECOND),
from <MY TEMP BIGQUERY TABLE> s  
inner join operations_sucess ss
    on EXTRACT(HOUR FROM s.timestamp) = ss.hour and ss.resource_name = s.resource_name 
where s.operation_type ='incomplete'
group by s.resource_name,EXTRACT(HOUR FROM s.timestamp)

output will be:

1| foo | 633
2| bar | 513
3| bar | 813

Please let me know if this code help you in your process.

Regards,