如何在 Spark Scala 中将列数据类型转换为字符串？

Question

I am trying to call the from_json method and want to fetch the schema of the JSON dynamically. The issue is with the third.withColumn line as it doesn't seem to like Seq[Column].

val randomStringGen = udf((length: Int) => {
scala.util.Random.alphanumeric.take(length).mkString
})

val randomKeyGen = udf((key: String, value: String) => {
s"""{"${key}": "${value}"}"""
})

val resultDF = initDF
.withColumn("value", randomStringGen(lit(10)))
.withColumn("keyValue", randomKeyGen(lit("key"), col("value")))
.withColumn("key", from_json(col("keyValue"), spark.read.json(Seq(col("keyValue")).toDS).schema))

error: value toDS is not a member of Seq[org.apache.spark.sql.Column].withColumn("key", from_json(col("keyValue"), spark.read.json(Seq(col("keyValue")).toDS).schema))

I have a known solution which is simply to hard code a sample JSON:

val jsData = """{"key": "value"}"""

and replace the col("keyValue") with the hardcoded variable.

.withColumn("key", from_json(col("keyValue"), spark.read.json(Seq(jsData).toDS).schema))

This works and produces exactly what I want, but if I have a large json, then this method can be quite cumbersome.

Answer 1

There are 2 little errors in what you're writing.

First, if you want to use the toDS method on a Seq you'll need to import spark.implicits._ . This method is defined in there. Doing that should get rid of your first error.

Secondly, the from_json function you're trying to use has the following function signature:

def from_json(e: Column, schema: StructType): Column

So that second field, schema , should be a StructType. This is what the .schema method of a Dataset returns. So, one of your parentheses is at the wrong position.

Instead of

.withColumn("key", from_json(col("keyValue"), spark.read.json(Seq(col("keyValue")).toDS.schema)))

you should have

.withColumn("key", from_json(col("keyValue"), spark.read.json(Seq(col("keyValue")).toDS).schema))