[英]TypeError: udf() missing 1 required positional argument: 'f'
I couldn't find any solution or question to my problem.我找不到我的问题的任何解决方案或问题。
If I try to define a Spark-UDF Function (pyspark) eg:如果我尝试定义 Spark-UDF Function (pyspark) 例如:
@udf(returnType=IntegerType())
def _return_an_integer(y):
return y + random.randint(1, 99)
df = df.withColumn('randomInteger', _return_an_integer(col('1')))
I get the error message:我收到错误消息:
TypeError: udf() missing 1 required positional argument: 'f'
Has anybody a solution for this?有没有人解决这个问题?
Thanks in advance!提前致谢!
EDIT: Without annotation:编辑:没有注释:
Py4JError: An error occurred while calling None.org.apache.spark.sql.execution.python.UserDefinedPythonFunction. Trace:
py4j.Py4JException: Constructor org.apache.spark.sql.execution.python.UserDefinedPythonFunction([class java.lang.String, class org.apache.spark.api.python.PythonFunction, class org.apache.spark.sql.types.IntegerType$]) does not exist
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:179)
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:196)
at py4j.Gateway.invoke(Gateway.java:237)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
After trying lot of things, the problem was that my pyspark version didn't match the spark version.在尝试了很多东西之后,问题是我的 pyspark 版本与 spark 版本不匹配。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.