[英]Wrapping a java function in pyspark
I am trying to create a user defined aggregate function which I can call from python. 我正在尝试创建一个可以从python调用的用户定义的聚合函数。 I tried to follow the answer to this question.
我试图遵循这个问题的答案。 I basically implemented the following (taken from here ):
我基本上实现了以下内容(从此处获取 ):
package com.blu.bla;
import java.util.ArrayList;
import java.util.List;
import org.apache.spark.sql.expressions.MutableAggregationBuffer;
import org.apache.spark.sql.expressions.UserDefinedAggregateFunction;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
import org.apache.spark.sql.types.DataType;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.Row;
public class MySum extends UserDefinedAggregateFunction {
private StructType _inputDataType;
private StructType _bufferSchema;
private DataType _returnDataType;
public MySum() {
List<StructField> inputFields = new ArrayList<StructField>();
inputFields.add(DataTypes.createStructField("inputDouble", DataTypes.DoubleType, true));
_inputDataType = DataTypes.createStructType(inputFields);
List<StructField> bufferFields = new ArrayList<StructField>();
bufferFields.add(DataTypes.createStructField("bufferDouble", DataTypes.DoubleType, true));
_bufferSchema = DataTypes.createStructType(bufferFields);
_returnDataType = DataTypes.DoubleType;
}
@Override public StructType inputSchema() {
return _inputDataType;
}
@Override public StructType bufferSchema() {
return _bufferSchema;
}
@Override public DataType dataType() {
return _returnDataType;
}
@Override public boolean deterministic() {
return true;
}
@Override public void initialize(MutableAggregationBuffer buffer) {
buffer.update(0, null);
}
@Override public void update(MutableAggregationBuffer buffer, Row input) {
if (!input.isNullAt(0)) {
if (buffer.isNullAt(0)) {
buffer.update(0, input.getDouble(0));
} else {
Double newValue = input.getDouble(0) + buffer.getDouble(0);
buffer.update(0, newValue);
}
}
}
@Override public void merge(MutableAggregationBuffer buffer1, Row buffer2) {
if (!buffer2.isNullAt(0)) {
if (buffer1.isNullAt(0)) {
buffer1.update(0, buffer2.getDouble(0));
} else {
Double newValue = buffer2.getDouble(0) + buffer1.getDouble(0);
buffer1.update(0, newValue);
}
}
}
@Override public Object evaluate(Row buffer) {
if (buffer.isNullAt(0)) {
return null;
} else {
return buffer.getDouble(0);
}
}
}
I then compiled it with all dependencies and run pyspark with --jars myjar.jar 然后,我使用所有依赖项对其进行编译,并使用--jars myjar.jar运行pyspark
In pyspark I did: 在pyspark我做了:
df = sqlCtx.createDataFrame([(1.0, "a"), (2.0, "b"), (3.0, "C")], ["A", "B"])
from pyspark.sql.column import Column, _to_java_column, _to_seq
from pyspark.sql import Row
def myCol(col):
_f = sc._jvm.com.blu.bla.MySum.apply
return Column(_f(_to_seq(sc,[col], _to_java_column)))
b = df.agg(myCol("A"))
I got the following error: 我收到以下错误:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-24-f45b2a367e67> in <module>()
----> 1 b = df.agg(myCol("A"))
<ipython-input-22-afcb8884e1db> in myCol(col)
4 def myCol(col):
5 _f = sc._jvm.com.blu.bla.MySum.apply
----> 6 return Column(_f(_to_seq(sc,[col], _to_java_column)))
TypeError: 'JavaPackage' object is not callable
I also tried adding --driver-class-path to the pyspark call but got the same result. 我也尝试将--driver-class-path添加到pyspark调用中,但是得到了相同的结果。
Also tried to access the java class through java import: 还尝试通过java import访问java类:
from py4j.java_gateway import java_import
jvm = sc._gateway.jvm
java_import(jvm, "com.bla.blu.MySum")
def myCol2(col):
_f = jvm.bla.blu.MySum.apply
return Column(_f(_to_seq(sc,[col], _to_java_column)))
Also Tried to simply create the class (as suggested here ): 还试图简单地创建类(如建议在这里 ):
a = jvm.com.bla.blu.MySum()
All are getting the same error message. 所有人都收到相同的错误消息。
I can't seem to figure out what the problem is. 我似乎无法弄清楚问题出在哪里。
So it seems the main issue was that all of the options to add the jar (--jars, driver class path, SPARK_CLASSPATH) do not work properly if giving a relative path. 因此,似乎主要的问题是,如果提供相对路径,则添加jar的所有选项(--jars,驱动程序类路径,SPARK_CLASSPATH)都无法正常工作。 THis is probably because of issues with the working directory inside ipython as opposed to where I ran pyspark.
这可能是因为ipython内的工作目录存在问题,而不是我在pyspark上运行的地方。
Once I changed this to absolute path, it works (Haven't tested it on a cluster yet but at least it works on a local installation). 一旦将其更改为绝对路径,它就可以使用(尚未在群集上进行测试,但至少可以在本地安装上使用)。
Also, I am not sure if this is a bug also in the answer here as that answer uses a scala implementation, however in the java implementation I needed to do 另外,我不知道这是一个错误也是在回答此作为答案的Java实现,我需要做使用Scala实现,然而,
def myCol(col):
_f = sc._jvm.com.blu.bla.MySum().apply
return Column(_f(_to_seq(sc,[col], _to_java_column)))
This is probably not really efficient as it creates _f each time, instead I should probably define _f outside the function (again, this would require testing on the cluster) but at least now it provides the correct functional answer 这可能并不真正有效,因为它每次都会创建_f,相反,我应该在函数外部定义_f(再次,这将需要在集群上进行测试),但至少现在它提供了正确的功能答案
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.