how to calculate the time length of 0-1 sequence with hive?

Question

Now I have a data like:

time(string) id(int)

201801051127 0

201801051130 0

201801051132 0

201801051135 1

201801051141 1

201801051145 0

201801051147 0

It has three different parts, and I want to calculate the time length of these three parts, such as the first zero sequence, the time length is 5 minutes. If I use 'group by 0 and 1', the first zero sequence would combine with the third zero sequence, which is not what I want. How I calculate the three parts' length with sql? My tried my-sql code is as follows:

SET @id_label:=0;
SELECT id_label,id,TIMESTAMPDIFF(MINUTE,MIN(DATE1),MAX(DATE1)) FROM
(SELECT id, DATE1, id_label FROM (
SELECT id, str_to_date ( TIME,'%Y%m%d%H%i' ) DATE1,
@id_label := IF(@id = id, @id_label, @id_label+1)  id_label,
@id := id
FROM test.t
ORDER BY str_to_date ( TIME,'%Y%m%d%h%i' )
) a)b
GROUP BY id_label,id;

I don't know how to change it into hive code.

Answer 1

Try This.

SELECT id, ( max( TO_DATE ( time,'YYYYMMDDHHMI' ) )
- min( TO_DATE ( time,'YYYYMMDDHHMI' ) ) ) *24*60 diff_in_minutes from 
(
select t.*,
row_number()   OVER ( ORDER BY 
                    TO_DATE ( time,'YYYYMMDDHHMI' ) )
- row_number() OVER ( PARTITION BY ID ORDER BY 
                    TO_DATE ( time,'YYYYMMDDHHMI' ) ) seq
FROM Table1 t ORDER BY time
  ) GROUP BY ID,seq
  ORDER BY max(time)
  ;

DEMO

EDIT: This answer was written considering that the OP had tagged oracle .Now it is changed to hive .

As an alternative in hive for TO_DATE in Oracle,

unix_timestamp(time, 'yyyyMMddhhmm') 

could be used.

Answer 2

I would suggest some transformations:

• add an indication whether a row is the first one in its group (flag as 1, or null otherwise)
• count the number of such flags that precede a row to know its group number

Then you can just group by that new group number.

Oracle version (original question)

with q1 as (
    select to_date(time, 'YYYYMMDDHH24MI') time, id, 
           case id when lag(id) over(order by time) then null else 1 end first_in_group 
    from t
), q2 as (
    select time, id, count(first_in_group) over (order by time) grp_id
    from   q1
)
select   min(id) id, (max(time) - min(time)) * 24 * 60 minutes
from     q2
group by grp_id
order by grp_id

SQL fiddle

Hive version

Different database engines use different functions to deal with date/time values, so use Hive's unix_timestamp and deal with the number of seconds it returns:

with q1 as (
    select unix_timestamp(time, 'yyyyMMddHHmm')/60 time, id, 
           case id when lag(id) over(order by time) then null else 1 end first_in_group 
    from t
), q2 as (
    select time, id, count(first_in_group) over (order by time) grp_id
    from   q1
)
select   min(id) id, max(time) - min(time) minutes
from     q2
group by grp_id
order by grp_id