缺少当前日期的30天滚动/移动总和

Question

I have a table ( view_of_referred_events ) which stores the number of visitors for a given page.

date        country_id  referral    product_id  visitors
2016-04-01  216         pl          113759      1
2016-04-03  216         pl          113759      1
2016-04-06  216         pl          113759      13
2016-04-07  216         pl          113759      10

I want to compute the 30-day rolling/moving sum for this product, even for those days which are missing. So the end result should be something like the following:

date        country_id  referral    product_id  cumulative_visitors
2016-04-01  216         pl          113759      1
2016-04-02  216         pl          113759      1
2016-04-03  216         pl          113759      2
2016-04-04  216         pl          113759      2
2016-04-05  216         pl          113759      2
2016-04-06  216         pl          113759      15
2016-04-07  216         pl          113759      25

Now, this is a simplistic representation, because I have tens of different country_id , referral and product_id . I can't pre-create a table with all possible combinations of { date , country_id , referral and product_id } because this would become untreatable considering the size of the table. I don't also want to have a row in the final table if that specific { date , country_id , referral and product_id } didn't exist before.

I was thinking if there was an easy way to tell Impala to use the value of the previous row (the previous day) if in view_of_referred_events there are no visitors for that day.

I wrote this query, where list_of_dates is a table with a list of days from April 1st to April 7th.

select
  t.`date`,
  t.country_id,
  t.referral,
  t.product_id,
  sum(visitors) over (partition by t.country_id, t.referral, t.product_id order by t.`date`
                     rows between 30 preceding and current row) as cumulative_sum_visitors
from (
  selec
    d.`date`, 
    re.country_id, 
    re.referral, 
    re.product_id,
    sum(visitors) as visitors
  from list_of_dates d
  left outer join view_of_referred_events re on d.`date` = re.`date`
    and re.referral = "pl"
    and re.product_id = "113759"
    and re.country_id = "216"
  group by d.`date`, re.country_id, re.referral, re.product_id
  ) t
order by t.`date` asc;

This returns something similar to what I want, but not exactly that.

date        country_id  referral    product_id  cumulative_visitors
2016-04-01  216         pl          113759      1
2016-04-02  NULL        NULL        NULL        NULL
2016-04-03  216         pl          113759      2
2016-04-04  NULL        NULL        NULL        NULL
2016-04-05  NULL        NULL        NULL        NULL
2016-04-06  216         pl          113759      15
2016-04-07  216         pl          113759      25

Answer 1

I'm not sure how goo the performance will be, but you can do this by aggregating the data twice and adding 30 days for the second aggregation and negating the count.

Something like this:

with t as (
      select d.`date`, re.country_id, re.referral, re.product_id,
             sum(visitors) as visitors
      from list_of_dates d left outer join
           view_of_referred_events re
           on d.`date` = re.`date` and
              re.referral = 'pl' and
              re.product_id = 113759 and
              re.country_id = 216
      group by d.`date`, re.country_id, re.referral, re.product_id
     )
select date, country_id, referral, product_id,
       sum(sum(visitors)) over (partition by country_id, referral, product_id order by date) as visitors
from ((select date, country_id, referral, product_id, visitors
       from t
      ) union all
      (select date_add(date, 30), country_id, referral, product_id, -visitors
       from t
      ) 
     ) tt
group by date, country_id, referral, product_id;

Answer 2

I have added another sub query to get the value from the last row in the partition. I am not sure what version of hive/impala you are using, last_value(column_name, ignore null values true/false) is the syntax.

I assume you are trying to find the cumulative counts over a 30 days (month), I recommend using month field to group the rows. The month could come either from your dimension table list_of_dates or just substr(date, 1, 7) and get the cumulative counts of visitors over ..rows unbounded preceding and current row .

query:

select
  `date`,
  country_id,
  referral,
  product_id,
  sum(visitors) over (partition by country_id, referral, product_id order by `date`
                     rows between 30 preceding and current row) as cumulative_sum_visitors 
from (select
  t.`date`,
  -- get the last not null value from the partition window w for country_id, referral & product_id
  last_value(t.country_id, true) over w as country_id,
  last_value(t.referral, true) over w as  referral
  last_value(t.product_id, true) over w as product_id 
  if(visitors = null, 0, visitors) as visitors 
from (
  select
    d.`date`, 
    re.country_id, 
    re.referral, 
    re.product_id,
    sum(visitors) as visitors
  from list_of_dates d
  left outer join view_of_referred_events re on d.`date` = re.`date`
    and re.referral = "pl"
    and re.product_id = "113759"
    and re.country_id = "216"
  group by d.`date`, re.country_id, re.referral, re.product_id
  ) t
window w as (partition by t.country_id, t.referral, t.product_id order by t.`date`
                     rows between unbounded preceding and unbounded following)) t1
order by `date` asc;