[英]Pyspark - Create new column with the RMSE of two other columns in dataframe
I am fairly new to Pyspark.我对 Pyspark 相当陌生。 I have a dataframe, and I would like to create a 3rd column with the calculation for RMSE between
col1
and col2
.我有一个 dataframe,我想创建一个第三列,计算
col1
和col2
之间的 RMSE。 I am using a user defined lambda function to make the RMSE calculation, but keep getting this error AttributeError: 'int' object has no attribute 'mean'
我正在使用用户定义的 lambda function 进行 RMSE 计算,但不断收到此错误
AttributeError: 'int' object has no attribute 'mean'
from pyspark.sql.functions import udf,col
from pyspark.sql.types import IntegerType
from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql import SQLContext
spark = SparkSession.builder.config("spark.driver.memory", "30g").appName('linear_data_pipeline').getOrCreate()
sqlContext = SQLContext(sc)
old_df = sqlContext.createDataFrame(sc.parallelize(
[(0, 1), (1, 3), (2, 5)]), ('col_1', 'col_2'))
function = udf(lambda col1, col2 : (((col1 - col2)**2).mean())**.5)
new_df = old_df.withColumn('col_n',function(col('col_1'), col('col_2')))
new_df.show()
How do I best go about fixing this issue?我如何最好地解决此问题? I would also like to find the RMSE/mean, mean absolute error, mean absolute error/mean, median absolute error, and Median Percent Error, but once I figure out how to calculate one, I should be good on the others.
我还想找到 RMSE/均值、平均绝对误差、平均绝对误差/均值、中值绝对误差和中值百分比误差,但是一旦我弄清楚如何计算一个,我应该会擅长其他的。
I think than you are some confused.我觉得比你还有些糊涂。 The RMSE is calculated from a succession of points, therefor you don't must calculate this for each value in two columns.
RMSE是根据一系列点计算得出的,因此您不必为两列中的每个值计算此值。 I think you have to calculate RMSE using all values in each column.
我认为您必须使用每列中的所有值来计算 RMSE。
This could works:这可能有效:
pow = udf(lambda x: x**2)
rmse = (sum(pow(old_df['col1'] - old_df['col2']))/len(old_df))**.5
print(rmse)
I don't think you need a udf
in that case.在这种情况下,我认为您不需要
udf
。 I think it is possible by using only pyspark.sql.functions
.我认为仅使用
pyspark.sql.functions
是可能的。
I can propose you the following untested option我可以向您推荐以下未经测试的选项
import pyspark.sql.functions as psf
rmse = old_df.withColumn("squarederror",
psf.pow(psf.col("col1") - psf.col("col2"),
psf.lit(2)
))
.agg(psf.avg(psf.col("squarederror")).alias("mse"))
.withColumn("rmse", psf.sqrt(psf.col("mse")))
rmse.collect()
Using the same logic, you can get other performance statistics使用相同的逻辑,您可以获得其他性能统计信息
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.