将具有文字值的新列添加到 Spark Scala 中 Dataframe 中的结构列

Question

我有一个 dataframe 具有以下架构：

root
 |-- docnumber: string (nullable = true)
 |-- event: struct (nullable = false)
 |    |-- data: struct (nullable = true)
           |-- codevent: int (nullable = true)

我需要在event.data中添加一列，以便架构如下所示：

root
 |-- docnumber: string (nullable = true)
 |-- event: struct (nullable = false)
 |    |-- data: struct (nullable = true)
           |-- codevent: int (nullable = true)
           |-- needtoaddit: int (nullable = true)

我试过了

• dataframe.withColumn("event.data.needtoaddit", lit("added"))

  但它添加了一个名为event.data.needtoaddit的列

• dataframe.withColumn( "event", struct( $"event.*", struct( lit("added").as("needtoaddit") ).as("data") ) )

  但它创建了一个名为event.data的模棱两可的列，我又遇到了问题。

我怎样才能让它工作？

Answer 1

你有点接近。 试试这个代码：

val df2 = df.withColumn(
    "event", 
    struct(
        struct(
            $"event.data.*", 
            lit("added").as("needtoaddit")
        ).as("data")
    )
)

Answer 2

火花 3.1+

要在结构列中添加字段，请使用withField

col("event.data").withField("needtoaddit", lit("added"))

---

输入：

val df = spark.createDataFrame(Seq(("1", 2)))
    .select(
        col("_1").as("docnumber"),
        struct(struct(col("_2").as("codevent")).as("data")).as("event")
    )
df.printSchema()
// root
//  |-- docnumber: string (nullable = true)
//  |-- event: struct (nullable = false)
//  |    |-- data: struct (nullable = false)
//  |    |    |-- codevent: long (nullable = true)

脚本：

val df2 = df.withColumn(
    "event",
    col("event.data").withField("needtoaddit", lit("added"))
)

df2.printSchema()
// root
//  |-- docnumber: string (nullable = true)
//  |-- event: struct (nullable = false)
//  |    |-- data: struct (nullable = true)
//            |-- codevent: int (nullable = true)
//            |-- needtoaddit: int (nullable = true)