PIG脚本如何

Question

I am trying clean up this employee volunteer data. There is no way to track if employee already is registered volunteer so he can sign up as new volunteer and will get a new VOLUNTEER_ID. I have a data feeding into where i can tie each VOLUNTEER_ID to its EMP_ID. The volunteer data needs to be cleaned up so we can figure out how the employee moved from a volunteer_level to another and when.

The business logic is that, when there is a overlaping dates, we give the highest level to the employee for the timeframe of between start_date and end_date.

I posted a Input sample of data and what the output should be.

Is it possible to do this a PIG script ? Can someone please help me

INPUT:

EMP_ID  VOLUNTEER_ID    V_LEVEL STATUS  START_DATE  END_DATE
10001   100               1      A       1/1/2006   12/31/2007
10001   200               1      A       5/1/2006   
10001   100               1      A       1/1/2008   
10001   300               3      P       3/1/2008   3/1/2008
10001   300               3      A       3/2/2008   12/1/2008
10001   1001              2      A       5/1/2008   6/30/2008
10001   1001              3      A       7/1/2008   
10001   300               2      A       12/2/2008  

OUTPUT NEEDED:( VOLUNTEER_ID is not needed in output but adding below to show which ID was selected for output and which did not)

EMP_ID  VOLUNTEER_ID    V_LEVEL STATUS  START_DATE  END_DATE
10001   100              1       A       1/1/2006   12/31/2007
10001   300              3       P       3/1/2008   3/1/2008
10001   300              3       A       3/2/2008   12/1/2008
10001   1001             2       A       5/1/2008   6/30/2008
10001   1001             3       A       7/1/2008   

Answer 1

It seems like you want the row in your data with the earliest start date for each V_LEVEL , STATUS , EMP_ID , and VOLUNTEER_ID

First we add a unix time column and then find the min for that column (this is in the latest version of pig so you may need to update your version).

data_with_unix = foreach data generate EMP_ID, VOLUNTEER_ID, V_LEVEL, STATUS, START_DATE, END_DATE, ToUnixTime((datetime)START_DATE) as unix_time;
grp = group data_with_unix by (EMP_ID, VOLUNTEER_ID, V_LEVEL, STATUS);
max_date = foreach grp generate group, MIN(data_with_unix.unix_time);

Then join the start and end date back into your dataset since there it doesn't look like there is currently a way to convert unix time back to date.