创建一个新的Spark数据集 <Row> 基于现有数据集 <Row> 和一个附加的HashMap

Question

I have a Dataset<Row> based on JSON-data. Now I would like to create a new Dataset<Row> based on the initial dataset BUT with a column added based on a Java HashMap<String, String> datatype something like

Dataset<Row> dataset2 = dataset1.withColumn("newColumn", *some way to specify HashMap<String, String> as the added column's datatype*);

Using this new dataset I could create a row-encoder such as

ExpressionEncoder<Row> dataset2Encoder = RowEncoder.apply(dataset2.schema());

and then apply a map-function such as

dataset2 = dataset2.map(new XyzFunction(), dataset2Encoder)

CLARIFICATION My initial dataset is based on data in JSON-format. What I'm trying to accomplish is to create a new dataset based on this initial dataset BUT with a new column added in the MapFunction. The thought of adding the column (withColumn) when creating the initial dataset would make sure that a schema definition would exist for the column I'd like to update in the MapFunction. However, I can't seem to find a way of modifying the Row object being passed to the call(Row arg) function of the MapFunction class OR create a new instance using RowFactory.create(...) in the call function. I'd like to be able to create a Row-instance in the MapFunction based on all the existing values of the passed Row-object AND a new Map to be added to the new row. The encoder would then know about this new/generated column from the generated schema. I hope this clarifies what I'm trying to accomplish...

Answer 1

You can

import static org.apache.spark.sql.functions.*;
import org.apache.spark.sql.types.DataTypes;

df.withColumn("newColumn", lit(null).cast("map<string, string>"));

or

df.withColumn(
  "newColumn", 
  lit(null).cast(
    DataTypes.createMapType(DataTypes.StringType, DataTypes.StringType)
  )
);

but why such indirection?

Encoder<Row> enc = RowEncoder.apply(df.schema().add(
  "newColumn",
  DataTypes.createMapType(DataTypes.StringType, DataTypes.StringType)
));

Depending on what you exactly try to do, using an UserDefinedFunction might much simpler, and allow you to skip Encoders completely.