Java中带有Spark Dataframe的直方图

Question

是否可以使用Java中的Spark 2.1从Dataset<Row>表生成直方图数据框？

Answer 1

1. Convert the Dataset into JavaRDD where Datatype can be Integer, Double etc. using toJavaRDD().map() function.
2. Again Convert the JavaRDD to JavaDoubleRDD using mapToDouble function.
3. Then you can apply histogram(int bucketcount) to get the histogram of the data.

Example : I got a table in spark with table name as 'nation' having column as 'n_nationkey' which is Integer then this is how I did it:

String query = "select n_nationkey from nation" ;
Dataset<Row> df = spark.sql(query);
JavaRDD<Integer> jdf = df.toJavaRDD().map(row -> row.getInt(0));
JavaDoubleRDD example = jdf.mapToDouble(y -> y);
Tuple2<double[], long[]> resultsnew = example.histogram(5);

In case the column have a double type, you simply replace some things as :

JavaRDD<Double> jdf = df.toJavaRDD().map(row -> row.getDouble(0));
JavaDoubleRDD example = jdf.mapToDouble(y -> y);
Tuple2<double[], long[]> resultsnew = example.histogram(5);