Jersey Streamingoutput of large data set

Question

final StreamingOutput stream = new StreamingOutput() {
  @Override
  public void write(final OutputStream out) {
  dao.getData(
    query,
    new SdmxObserver(writerFactory.getDataWriter(sdmxFormat, out, properties), request
        .getRemoteAddr(), request.getHeader("User-Agent"), query.toString(), sdmxFormat
        .toString(), query.getlist()));
  }
};
res = Response.ok(stream, MediaType.valueOf("application/vnd.sdmx.genericdata+xml;version=2.1"))
            .cacheControl(cc).lastModified(lastModified).header("Vary", "Accept,Accept-Encoding").build();
return res;

The database call to retrieve the data is taking long as the data is huge and for that before downloading the data in the browser it is taking long time and user thought the server is not reachable. How could we send the chunk of data by using multiple database query in the same web service response, so that the download starts quickly and keep adding the response when next query fetch the data from database.

Answer 1

Paginate the response content into pages up to X records. This will allow the API to retrieve a single page of results at a time, one by one.

http://localhost/api/v1/resource?page=1&size=20
http://localhost/api/v1/resource?page=2&size=20
http://localhost/api/v1/resource?page=3&size=20
...

You can take a look at PageRequest and Pageable defined in Spring Data when defining your API.