in sparkR API there are functions with the same name as in R. Some of the examples are abs,cosine
functions.
What is the difference between abs function in R and in sparkR. when does the abs function get executed in spark?
documentation for sparkR abs function http://spark.apache.org/docs/latest/api/R/abs.html
The difference is where the function lives.
In base R, abs
is a primitive:
function(x) .Primitive("abs")
In Spark, abs
is a wrapper around a call to the Spark engine:
setMethod("abs",
signature(x = "Column"),
function(x) {
jc <- callJStatic("org.apache.spark.sql.functions", "abs", x@jc)
column(jc)
})
You can see the R source code for the SparkR package here .
In base R,it can be applied to any vector, but in SparkR, it can only be applied to columns.Suppose you have a dataframe and C3 column is double, you can use following code to add one column C4 to dataframe, which is the abs value for C3.
df$C4 <- abs(df$C3)
or
withColumn(df,"absvalue",abs(df&C3))
I think the most difference between base R and SparkR is that in SparkR, the min unit you can conduct is column, not as vector or matrix. I am just a newcomer for SparkR, I am still learning.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.