如果Spark SQL支持HQL，例如“插入覆盖目录”？

Question

I would like to ask if Spark SQL support HQL like 'insert overwrite directory'. Or is there another way to save the Result set（from a spark sql jdbc server） to HDFS directly?

Answer 1

There is one jira for this issue which is not yet resolved the jira link for that is https://issues.apache.org/jira/browse/SPARK-4131 . But you can do something like this:

    JavaSchemaRDD employeeSchemaRDD = context.sql("SELECT * FROM employee");

    JavaRDD<String> strRDD=employeeSchemaRDD.map(new Function<Row, String>() {

        public String call(Row row) throws Exception {
            // TODO Auto-generated method stub
            return row.get(1).toString();
        }

    });
strRDD.saveAsTextFile("outputdir");

Replace outputdir with HDFS url where you want to write the output. Hope this answer you question.