使用猪计算一组中的不同值

Question

一般而言，我的问题是我想对数据进行分组，然后计算字段的uniq值。 具体来说，对于以下数据，我想按“类别”和“年份”分组，然后计算“食物”的uniq值。

category,id,mydate,mystore,food    
catA,myid_1,2014-03-11 13:13:13,store1,apple
catA,myid_2,2014-03-11 12:12:12,store1,milk
catA,myid_3,2014-08-11 10:13:13,store1,apple
catA,myid_4,2014-09-11 09:12:12,store1,milk
catA,myid_5,2015-09-01 10:10:10,store1,milk
catB,myid_6,2014-03-12 03:03:03,store2,milk
catB,myid_7,2014-03-12 05:55:55,store2,apple

据我所知，这只是挑选值并使用一些简洁的Pig日期函数：

a = load '$input' using PigStorage(',') as (category:chararray,id:chararray,mydate:chararray,mystore:chararray,food:chararray);
b = foreach a generate category, id, ToDate(mydate,'yyyy-MM-dd HH:mm:ss') as myDt:DateTime, mystore,food;


c = foreach b generate category, GetYear(myDt) as year:int, mystore,food;      
dump c;

别名“ c”的输出为：

(catA,2014,store1,apple)
(catA,2014,store1,milk)
(catA,2014,store1,apple)
(catA,2014,store1,milk)
(catA,2015,store1,milk)
(catB,2014,store2,milk)
(catB,2014,store2,apple)

最后我要：

catA, 2014, {(apple, 2), (milk, 2)} 
catA, 2015, {(milk, 1)} 
catB, 2014, {(apple, 1), (milk, 1)} 

我已经看到了一些产生价值计数的示例，但是按类别和年份分组使我感到震惊。

Answer 1

输入：

类别，标识，指明MyDate，的MyStore，食品

catA,myid_1,2014-03-11 13:13:13,store1,apple
catA,myid_2,2014-03-11 12:12:12,store1,milk
catA,myid_3,2014-08-11 10:13:13,store1,apple
catA,myid_4,2014-09-11 09:12:12,store1,milk
catA,myid_5,2015-09-01 10:10:10,store1,milk
catB,myid_6,2014-03-12 03:03:03,store2,milk
catB,myid_7,2014-03-12 05:55:55,store2,apple

是的，您可以在分组后使用嵌套的FOREACH，在该嵌套的FOREACH中，您可以对食物应用Distinct，然后可以对其进行计数。

下面的代码将帮助您

猪脚本：

list = LOAD 'user/cloudera/apple.txt' USING PigStorage(',') AS(category:chararray,id:chararray,mydate:chararray,my_store:chararray,food:chararray);

list_each = FOREACH list GENERATE category,SUBSTRING(mydate,0,4) as my_year, my_store, food;

list_grp = GROUP list_each BY (category,my_year);

list_nested_each = FOREACH list_grp

                            {
                               list_inner_each = FOREACH list_each GENERATE food;
                               list_inner_dist = DISTINCT list_inner_each;

                             GENERATE flatten(group) as (catgeory,my_year), COUNT(list_inner_dist) as no_of_uniq_foods;

                            };

dump list_nested_each;

输出：

(catA,2014,2)
(catA,2015,1)
(catB,2014,2)

Answer 2

附加到问题中的代码：

d = group c by (category, year, food);
e = foreach d generate FLATTEN(group), COUNT(c) as count;

将产生：

(catA,2014,milk,2)
(catA,2014,apple,2)
(catA,2015,milk,1)
(catB,2014,milk,1)
(catB,2014,apple,1)

关键是也要按“食物”分组。 有趣。 任何其他见解都受到欢迎。