Scala spark + 编码器问题

Question

Working on a problem where I need to add a new column that holds the length of all characters under all columns.

My sample data set :

ItemNumber,StoreNumber,SaleAmount,Quantity, Date
2231      ,  1        , 400      ,  2     , 19/01/2020
2145      ,  3        , 500      ,  10    , 14/01/2020

The expected output would be

19 20

The ideal output am expecting to build is with new column Length added to the data frame

ItemNumber,StoreNumber,SaleAmount,Quantity, Date      , Length
2231      ,  1        , 400      ,  2     , 19/01/2020, 19
2145      ,  3        , 500      ,  10    , 14/01/2020, 20

My code

 val spark = SparkSession.builder()
    .appName("SimpleNewIntColumn").master("local").enableHiveSupport().getOrCreate()

  val df = spark.read.option("header","true").csv("./data/sales.csv")

  var schema = new StructType

  df.schema.toList.map{
    each => schema = schema.add(each)
  }
  val encoder = RowEncoder(schema)

  val charLength = (row :Row) => {
    var len :Int = 0
    row.toSeq.map(x =>  {
      x match {
        case a : Int => len = len + a.toString.length
        case a : String => len = len + a.length
      }

    })
    len
  }

  df.map(row => charLength(row))(encoder) // ERROR - Required Encoder[Int] Found EncoderExpression[Row]

  df.withColumn("Length", ?)

I have two issues

1) How to solve the error "ERROR - Required Encoder[Int] Found EncodeExpression[Row]"?

2) How do I add the output of charLength function as new column value? - df.withColumn("Length", ?)

Thank you.

Gurupraveen

Answer 1

If you are just trying to add a column, with total length of that Row

You can simply concat all the columns cast to String and use length function

import org.apache.spark.sql.functions._
import org.apache.spark.sql.types.StringType

val concatCol = concat(df.columns.map(col(_).cast(StringType)):_*)

df.withColumn("Length", length(concatCol))

Output:

+----------+-----------+----------+--------+----------+------+
|ItemNumber|StoreNumber|SaleAmount|Quantity|      Date|length|
+----------+-----------+----------+--------+----------+------+
|      2231|          1|       400|       2|19/01/2020|    19|
|      2145|          3|       500|      10|14/01/2020|    20|
+----------+-----------+----------+--------+----------+------+