I have dataframe schema -
resultDF.printSchema
|-- SKU_ID_MAP: string (nullable = true)
|-- SKU_IMAGE_MAP: map (nullable = true)
| |-- key: string
| |-- value: struct (valueContainsNull = true)
| | |-- image_id: string (nullable = true)
| | |-- image_name: string (nullable = true)
| | |-- image_path: string (nullable = true)
I want to create a final dataframe like this from above DF.
case class Devicesku2 (
sku_id : String,
sku_images: Map[String, ImageInfo2]
)
resultDF.map(
row => Devicesku2(
row.getAs[String]("SKU_ID"),
row.getAs[Map]("SKU_IMAGE_MAP")
).toDF
In above row.getAs[Map] is giving compile time error as value is struct type.
Can someone help on this? Thanks,
`
If you rename your case class
's elements to:
case class Devicesku2 (
sku_id_map: String,
sku_image_map: Map[String, ImageInfo2]
)
You can just use
resultDF.as[Devicesku2]
Otherwise, as mentioned in Aluan Haddad's comment, you'll need
row.getAs[Map[String, ImageInfo2]]("SKU_IMAGE_MAP")
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.