Get a JSON from a Dataset in Spark SQL (java)

Question

I have a Spark SQL application running on a server. It takes data from .parquet files and in each request performs an SQL query on those data. I need to send the JSON corresponding to the output of the query in the response.

This is what I do

Dataset<Row> sqlDF = spark.sql(query);
sqlDF.show();

So I know that the query works.

I tried returning sqlDF.toJSON().collect() , but in the other end I only receive [Ljava.lang.String;@1cd86ff9 .

I tried writing sqlDF as a JSON file, but then I don't know how to add its content to the response, and it saves a structure of files that have nothing to do with a JSON file.

Any idea/suggestion?

Answer 1

You can return JSON String using the below code.
List<String> stringDataset = sqlDF.toJSON().collectAsList();
return stringDataset;

Jackson will return the JSON string in this case.

If you want to return proper JSONObject the you can use the below code :

List<Map<String,Object>> result= new ArrayList<>();
List<String> stringDataset = sqlDF.toJSON().collectAsList();
for(String s : stringDataset){
   Map<String,Object> map = new HashMap<>();
   map = mapper.readValue(s, new TypeReference<Map<String, String>>(){});
   result.add(map);
             }