[英]Merging Multiple Relation in PIG
Hi guys I am trying to crack this out i wonder if there's a file with attributes like this: 嗨,大家好,我想破解这个问题,我想知道是否有一个文件具有这样的属性:
(id#123, event#sasa, value#abcde, time#213, userid#21321)
to get total data i'd do: 获得我会做的总数据:
data_count = foreach (group data all) generate count(data);
to get total user i'd do: 为了获得总用户数,我愿意这样做:
group_users = GROUP data BY userid;
grp_all = GROUP group_users ALL;
count_users = FOREACH grp_all GENERATE COUNT(group_users);
now i wonder how do i merge these into 1 file that outputs 现在我想知道如何将这些合并到1个输出的文件中
(id, event, value, time, total data,total users)
thanks a lot. 非常感谢。
Not sure what is total data, but if you want to get back to the original rows with the total user count, than you need to use FLATTEN a couple of times. 不确定什么是总数据,但是如果您想返回具有总用户数的原始行,则需要使用FLATTEN几次。 PIG is not SQL, it works on BAGs and FLATEN converts BAGs back to rows.
PIG不是SQL,它可用于BAG,而FLATEN可将BAG转换回行。 For example:
例如:
data = load './data.csv' using PigStorage(',') as (e_id, e_name,value,time,userid);
group_users = GROUP data BY userid;
grp_all = GROUP group_users ALL;
DESCRIBE grp_all;
-- grp_all: {group: chararray,group_users: {(group: bytearray,data: {(e_id: bytearray,e_name: bytearray,value: bytearray,time: bytearray,userid: bytearray)})}}
uniq_users = FOREACH grp_all GENERATE FLATTEN(group_users), COUNT(group_users) as total_users;
describe uniq_users;
-- uniq_users: {group_users::group: bytearray,group_users::data: {(e_id: bytearray,e_name: bytearray,value: bytearray,time: bytearray,userid: bytearray)},total_users: long}
original = FOREACH uniq_users GENERATE FLATTEN(data), total_users;
describe original;
-- original: {group_users::data::e_id: bytearray,group_users::data::e_name: bytearray,group_users::data::value: bytearray,group_users::data::time: bytearray,group_users::data::userid: bytearray,total_users: long}
DUMP original;
i did it by using this script: 我通过使用以下脚本来做到这一点:
d1 = LOAD 'data' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad') AS (json:map[]);
d2 = foreach d1 generate
json#'event' AS EVENT,
json#'params'#'uid' AS USER,
ToDate(((long)json#'ts')*1000) AS DATE;
grpd = group d2 by EVENT;
uniq2 = foreach grpd {
usr = d2.USER;
unq_usr = distinct usr;
generate group,
d2.DATE,
COUNT(d2.EVENT),
COUNT(unq_usr);
};
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.