Pig中各个字段的值计数

Question

我有以下测试数据。

A   B   C

M   O

M   M   M

M   M   M

N       O

P       N

我想获取此样本测试数据中条目的总数，即12

我有下面的代码做同样的事情，但是我得到了错误的结果。

关于如何纠正的任何帮助将是有帮助的。

test=  LOAD 'testdata' USING PigStorage(',') as (A:chararray,B:chararray,C:chararray); 
values = FOREACH test GENERATE A==''?'null':(A is null?'null':A)) as A,(B==''?'null':(B is null?'null':B)) as B,(C==''?'null':(C is null?'null':C)) as C;  
grp = GROUP values ALL;  
counting = FOREACH grp GENERATE group, COUNT(values.A)+COUNT(values.B)+COUNT(values.C); 

这给出的答案是15而不是12。

我也想获取每个值的计数，例如M = 7，N = 2，O = 2，P = 1。 我写了下面的代码。

test=  LOAD 'testdata' USING PigStorage(',') as (A:chararray,B:chararray,C:chararray); 
values = FOREACH test GENERATE A==''?'null':(A is null?'null':A)) as A,(B==''?'null':(B is null?'null':B)) as B,(C==''?'null':(C is null?'null':C)) as C;  
grp = GROUP values ALL;  
    A = FOREACH grp {
B =FILTER test.A=='M' OR test.B=='M' OR test.C=='M';
GENERATE group, COUNT(B);
};

我收到一个错误“标量在输出中有多行”。

Answer 1

您还要在最终计数中计算列名。修改脚本以忽略第一行，然后按和计数。

test=  LOAD 'testdata' USING PigStorage(',') as (A:chararray,B:chararray,C:chararray); 

ranked = rank test;
test1 = Filter ranked by ($0 > 1); --Note:rank_test should work.

values = FOREACH test1 GENERATE A==''?'null':(A is null?'null':A)) as A,(B==''?'null':(B is null?'null':B)) as B,(C==''?'null':(C is null?'null':C)) as C;  
grp = GROUP values ALL;  
counting = FOREACH grp GENERATE group, COUNT(values.A)+COUNT(values.B)+COUNT(values.C);