将 Java 中的地图列表转换为 spark 中的数据集

Question

I have a list of Map in java, essentially representing rows.

List<Map<String, Object>> dataList = new ArrayList<>();
Map<String, Object> row1 = new HashMap<>();
row1.put("fund", "f1");
row1.put("broker", "b1");
row1.put("qty", 100);

Map<String, Object> row2 = new HashMap<>();
row2.put("fund", "f2");
row2.put("broker", "b2");
row2.put("qty", 200);

dataList.add(row1);
dataList.add(row2);

I'm trying to create a Spark DataFrame from it.

I've tried to convert it into JavaRDD<Map<String, Object>> using

JavaRDD<Map<String,Object>> rows = sc.parallelize(dataList);

But I'm not sure how to go from here to Dataset<Row> . I've seen Scala examples but none in Java.

I also tried to convert the list to JSON string, and read the JSON string.

String jsonStr = mapper.writeValueAsString(dataList);

But seems like I will have to write it to a file to then read using

Dataset<Row> df = spark.read().json(pathToFile);

I would prefer to do it in-memory if possible rather than write to file and read from there.

SparkConf sparkConf = new SparkConf().setAppName("SparkTest").setMaster("local[*]")
            .set("spark.sql.shuffle.partitions", "1");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
    SparkSession sparkSession = 
SparkSession.builder().config(sparkConf).getOrCreate();

List<Map<String, Object>> dataList = new ArrayList<>();
Map<String, Object> row1 = new HashMap<>();
row1.put("fund", "f1");
row1.put("broker", "b1");
row1.put("qty", 100);

Map<String, Object> row2 = new HashMap<>();
row2.put("fund", "f2");
row2.put("broker", "b2");
row2.put("qty", 200);

dataList.add(row1);
dataList.add(row2);

ObjectMapper mapper = new ObjectMapper();
    
String jsonStr = mapper.writeValueAsString(dataList);
JavaRDD<Map<String,Object>> rows = sc.parallelize(dataList);
Dataset<Row> data = sparkSession.createDataFrame(rows, Map.class);
data.show();

Answer 1

You do not need to use RDDs at all. What you need to do is extract the desired schema from your list of maps, transform you list of maps into a list of rows and then use spark.createDataFrame .

In java, that's a bit painful, particularly when creating the Row objects, but here is how it could go:

List<String> cols = new ArrayList(dataList.get(0).keySet());
List<Row> rows = dataList
    .stream()
    .map(row -> cols.stream().map(c -> (Object) row.get(c).toString()))
    .map(row -> row.collect(Collectors.toList()))
    .map(row -> JavaConverters.asScalaBufferConverter(row).asScala().toSeq())
    .map(Row$.MODULE$::fromSeq)
    .collect(Collectors.toList());

StructType schema = new StructType(
    cols.stream()
        .map(c -> new StructField(c, DataTypes.StringType, true, new Metadata()))
        .collect(Collectors.toList())
        .toArray(new StructField[0])
);
Dataset<Row> result = spark.createDataFrame(rows, schema);

Answer 2

public class MyRow implements Serializable {

  private String fund;
  private String broker;
  private int qty;

  public MyRow(String fund, String broker, int qty) {
    super();
    this.fund = fund;
    this.broker = broker;
    this.qty = qty;
  }

  public String getFund() {
    return fund;
  }

  public void setFund(String fund) {
    this.fund = fund;
  }


  public String getBroker() {
    return broker;
  }

  public void setBroker(String broker) {
    this.broker = broker;
  }

  public int getQty() {
    return qty;
  }

  public void setQty(int qty) {
    this.qty = qty;
  }

}

Now create an ArrayList. Each item in this list will act as row in final dataframe.

MyRow r1 = new MyRow("f1", "b1", 100);
MyRow r2 = new MyRow("f2", "b2", 200);
List<MyRow> dataList = new ArrayList<>();
dataList.add(r1);
dataList.add(r2);

Now We have to convert this List into a DataSet -

Dataset<Row> ds = spark.createDataFrame(dataList, MyRow.class);
ds.show()

Answer 3

The spark document had already point out how to load in-memory json string.

Here is the example fromhttps://spark.apache.org/docs/latest/sql-data-sources-json.html

// Alternatively, a DataFrame can be created for a JSON dataset represented by
// a Dataset<String> storing one JSON object per string.
List<String> jsonData = Arrays.asList(
        "{\"name\":\"Yin\",\"address\":{\"city\":\"Columbus\",\"state\":\"Ohio\"}}");
Dataset<String> anotherPeopleDataset = spark.createDataset(jsonData, Encoders.STRING());
Dataset<Row> anotherPeople = spark.read().json(anotherPeopleDataset);
anotherPeople.show();
// +---------------+----+
// |        address|name|
// +---------------+----+
// |[Columbus,Ohio]| Yin|
// +---------------+----+

May this help you.

Answer 4

import org.apache.spark.api.java.function.Function;
private static JavaRDD<Map<String, Object>> rows;
private static final Function f = (Function<Map<String, Object>, Row>) strObjMap -> RowFactory.create(new TreeMap<String, Object>(strObjMap).values().toArray(new Object[0]));
public void test(){
    rows = sc.parallelize(list);
    JavaRDD<Row> rowRDD = rows.map(f);
    Map<String, Object> headMap = list.get(0);
    TreeMap<String, Object> headerMap = new TreeMap<>(headMap);
    List<StructField> fields = new ArrayList<>();
    StructField field;
    for (String key : headerMap.keySet()) {
        System.out.println("key:::"+key);
        Object value = list.get(0).get(key);
        if (value instanceof Integer) {
            field = DataTypes.createStructField(key, DataTypes.IntegerType, true);
        }
        else if (value instanceof Double) {
            field = DataTypes.createStructField(key, DataTypes.DoubleType, true);
        }
        else if (value instanceof Date || value instanceof java.util.Date) {
            field = DataTypes.createStructField(key, DataTypes.DateType, true);
        }
        else {
            field = DataTypes.createStructField(key, DataTypes.StringType, true);
        }
            fields.add(field);
    }
    StructType struct = DataTypes.createStructType(fields);
    Dataset<Row> data = this.spark.createDataFrame(rowRDD, struct);
}