"Sparklyr 中的 Concat_ws() 函数丢失"

Question

I am following a tutorial on web (Adobe) analytics, where I want to build a Markov Chain Model. ( http:\/\/datafeedtoolbox.com\/attribution-theory-the-two-best-models-for-algorithmic-marketing-attribution-implemented-in-apache-spark-and-r\/<\/a> ).

Answer 1

You can simply use paste from base R.

library(sparklyr)
library(dplyr)

config <- spark_config()
sc <- spark_connect(master = "local", config = config)

df <- as.data.frame(cbind(c("1", "2", "3"), c("a", "b", "c")))
sdf <- sdf_copy_to(sc, df, overwrite = T)

sdf %>%
  mutate(concat = paste(V1, V2, sep = "-"))

Answer 2

You cannot find the function because it doesn't exist in sparklyr package. concat_ws is a Spark SQL function ( org.apache.spark.sql.functions.concat_ws ).

sparklyr depends on a SQL translation layer - function calls are translated into SQL expressions with dbplyr :

> dbplyr::translate_sql(concat_ws("-", foo, bar))
<SQL> CONCAT_WS('-', "foo", "bar")

This means that the function can be applied only in the sparklyr context:

sc <- spark_connect(master = "local[*]")
df <- copy_to(sc, tibble(x="foo", y="bar"))

df %>% mutate(xy = concat_ws("-", x, y))
# # Source: spark<?> [?? x 3]
#   x     y     xy     
# * <chr> <chr> <chr>  
# 1 foo   bar   foo-bar

Answer 3

I had a similar problem with dbplyr (BigQuery database).