在Spark Dataset API上使用汇总和分组依据

Question

    JavaRDD<Person> prdd = sc.textFile("c:\\fls\\people.txt").map(
          new Function<String, Person>() {
                public Person call(String line) throws Exception {
                  String[] parts = line.split(",");
                  Person person = new Person();
                  person.setName(parts[0]);
                  person.setAge(Integer.parseInt(parts[1].trim()));
                 person.setSal(Integer.parseInt(parts[2].trim()));
                  return person;
                }
              });

       RDD<Person>personRDD =  prdd.toRDD(prdd);
       Dataset<Person> dss= sqlContext.createDataset(personRDD ,               Encoders.bean(Person.class));
   GroupedDataset<Row, Person> dq=dss.groupBy(new Column("name"));

I have to calculate sum of age and salary group by name on the dataset. Please help how to query dataset ? I tried using GroupedDataset but don't know how to proceed with it. Thanks

Answer 1

You can register the JavaRDD prdd as a table and then used in SQL statements

`DataFrame schemaPeople = sqlContext.createDataFrame(prdd, Person.class); schemaPeople.registerTempTable("people");

// SQL can be run over RDDs that have been registered as tables.

DataFrame teenagers = sqlContext.sql("SELECT sum(age),sum(salary) FROM people group by name)

// The results of SQL queries are DataFrames and support all the normal RDD operations.`

Read more : http://spark.apache.org/docs/latest/sql-programming-guide.html#running-sql-queries-programmatically