简体   繁体   中英

what is the difference between 'abs' function in R and sparkR

in sparkR API there are functions with the same name as in R. Some of the examples are abs,cosine functions.

What is the difference between abs function in R and in sparkR. when does the abs function get executed in spark?

documentation for sparkR abs function http://spark.apache.org/docs/latest/api/R/abs.html

The difference is where the function lives.

In base R, abs is a primitive:

function(x) .Primitive("abs")

In Spark, abs is a wrapper around a call to the Spark engine:

setMethod("abs",
          signature(x = "Column"),
          function(x) {
            jc <- callJStatic("org.apache.spark.sql.functions", "abs", x@jc)
            column(jc)
          })

You can see the R source code for the SparkR package here .

In base R,it can be applied to any vector, but in SparkR, it can only be applied to columns.Suppose you have a dataframe and C3 column is double, you can use following code to add one column C4 to dataframe, which is the abs value for C3.

df$C4 <- abs(df$C3)

or

withColumn(df,"absvalue",abs(df&C3))

I think the most difference between base R and SparkR is that in SparkR, the min unit you can conduct is column, not as vector or matrix. I am just a newcomer for SparkR, I am still learning.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM