[英]Hive UDF : Hive does not send proper arguments to UDF
Here is my hive table这是我的 hive 表
CREATE TABLE `dum`(`val` map<string,array<string>>);
insert into dum select map('A',array('1','2','3'),'B',array('4','5','6'));
and here is how it looks这就是它的外观
select * from dum;
{"A":["1","2","3"],"B":["4","5","6"]}
I am trying to create a simple UDF that can combine all the items in the values of the above map into a list.我正在尝试创建一个简单的 UDF,它可以将上述 map 的值中的所有项目组合成一个列表。 Here is what i want to see
这是我想看到的
select modudf(val) from dum;
["1","2","3","4","5","6"]
so i created所以我创造了
package some.package;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.hive.ql.udf.UDFType;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
@UDFType(deterministic = true)
public class CustomUDF extends UDF {
public List<String> evaluate(Map<String, String[]> inMap) {
ArrayList<String> res = new ArrayList<String>();
for(Map.Entry<String, String[]> ent : inMap.entrySet()){
for(String item : ent.getValue())
res.add(item);
}
return res;
}
}
but when i try to invoke it as但是当我尝试调用它时
add jar /path/to/my/jar;
CREATE TEMPORARY FUNCTION modudf AS 'some.package.CustomUDF';
select modudf(val) from dum;
i get我明白了
FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments 'val': No matching method for class some.package.CustomUDF with (map<string,array<string>>). Possible choices: _FUNC_(map<struct<>,struct<>>)
why does hive think that my UDF requires map<struct<>,struct<>>
instead of map<string,array<string>>
?为什么 hive 认为我的 UDF 需要
map<struct<>,struct<>>
而不是map<string,array<string>>
? I even tried replacing String with Charsequence but i got the same error我什至尝试用 Charsequence 替换 String 但我得到了同样的错误
Note that as per the documentation请注意,根据文档
https://hive.apache.org/javadocs/r1.2.2/api/org/apache/hadoop/hive/ql/exec/UDF.html https://hive.apache.org/javadocs/r1.2.2/api/org/apache/hadoop/hive/ql/exec/UDF.ZFC35FDC70D5FC69D2639883A822CA
I should be able to use collections as input to the evaluate
method我应该能够使用 collections 作为
evaluate
方法的输入
What am i doing wrong?我究竟做错了什么?
Update更新
I also tried the following definition我还尝试了以下定义
public List<CharSequence> evaluate(Map<CharSequence, List<CharSequence>> inMap) {
modLogger.info(inMap);
ArrayList<CharSequence> res = new ArrayList<CharSequence>();
for(Map.Entry<CharSequence, List<CharSequence>> ent : inMap.entrySet()){
for(CharSequence item : ent.getValue())
res.add(item);
}
return res;
}
}
but i still get但我仍然得到
hive> add jar /path/to/my/jar;
Added [/path/to/my/jar] to class path
Added resources: [/path/to/my/jar]
hive> CREATE TEMPORARY FUNCTION modudf AS 'some.package.CustomUDF';
hive> desc dum;
OK
val map<string,array<string>>
Time taken: 0.094 seconds, Fetched: 1 row(s)
hive> select val from dum;
Query ID = root_20200629170147_80b5248f-4519-4dae-a070-3c5185f742ea
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1593449512239_0001)
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0 0 0
----------------------------------------------------------------------------------------------
VERTICES: 01/01 [==========================>>] 100% ELAPSED TIME: 6.12 s
----------------------------------------------------------------------------------------------
OK
{"A":["1","2","3"],"B":["4","5","6"]}
Time taken: 10.631 seconds, Fetched: 1 row(s)
hive> select modudf(val) from dum;
FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments 'val': No matching method for class com.walmart.labs.search.sib.gcp.ModularTransformUDF with (map<string,array<string>>). Possible choices: _FUNC_(map<struct<>,array<struct<>>>)
See the citation from the link you've sent :查看您发送的链接中的引文:
Note that Hive Arrays are represented as Lists in Hive.
请注意,Hive Arrays 在 Hive 中表示为列表。 So an ARRAY column would be passed in as a List.
因此,ARRAY 列将作为列表传入。
So you should have evaluate(Map<String, List<String>> inMap)
signature instead of evaluate(Map<String, String[]> inMap)
.所以你应该有
evaluate(Map<String, List<String>> inMap)
签名而不是evaluate(Map<String, String[]> inMap)
。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.