How to write java.util.stream.Stream to StreamingResponseBody output stream

Question

I am working to build a REST API where a large amount of data from the Oracle database can be sent in chunks via streaming to client application (like a file download or direct stream).

I am getting Stream from JpaRepository as given below -

@Query("select u from UsersEntity u")
Stream<UsersEntity> findAllByCustomQueryAndStream();

But now challenge comes to write this stream to StreamingResponseBody Output stream

I tried by many ways but no success -

First Approach -

Stream<UsersEntity> usersResultStream = usersRepository.findAllByCustomQueryAndStream();

        StreamingResponseBody stream = outputStream -> {
            Iterator<UsersEntity> iterator = usersResultStream.iterator();

            try (ObjectOutputStream oos = new ObjectOutputStream(outputStream)) {

                while (iterator.hasNext()) {
                    oos.write(iterator.next().toString().getBytes());
                }
            }
        };

Got Error -

java.sql.SQLException: Closed Resultset: next
    at oracle.jdbc.driver.InsensitiveScrollableResultSet.next(InsensitiveScrollableResultSet.java:565) ~[ojdbc7-12.1.0.2.jar:12.1.0.2.0]

Second Approach -

StreamingResponseBody stream = new StreamingResponseBody() {

            @Transactional(readOnly = true)
            @Override
            public void writeTo(OutputStream outputStream) throws IOException {

                Stream<UsersEntity> usersResultStream = usersRepository.findAllByCustomQueryAndStream();

                try (ObjectOutputStream oos = new ObjectOutputStream(outputStream)) {

                    usersResultStream.forEach(user->{
                        try {
                            oos.write(user.toString().getBytes());
                        } catch (IOException e) {
                            e.printStackTrace();
                        }
                    });
                }
            }
        }; 

Got Error -

org.springframework.dao.InvalidDataAccessApiUsageException: You're trying to execute a streaming query method without a surrounding transaction that keeps the connection open so that the Stream can actually be consumed. Make sure the code consuming the stream uses @Transactional or any other way of declaring a (read-only) transaction.

I have uploaded practice code at the below-given link - Sample POC Link

I don't have any experience with the streaming related task so please help me with this.

If I am in the wrong direction than suggest any other approach to do this within Spring Framework . Please share any reference links if available.

Answer 1

Finally, I resolved the problem by using the service layer. Initially, I was writing the complete logic in Controller Class which was creating the issue.

Controller Class -

@RestController
@RequestMapping("/api")
public class UsersController {
    @Autowired
    private UserService service;

    @GetMapping(value = "/userstream")
    public ResponseEntity<StreamingResponseBody> fetchUsersStream() {

        StreamingResponseBody stream = this::writeTo;

        return new ResponseEntity<>(stream, HttpStatus.OK);
    }

    private void writeTo(OutputStream outputStream) {
        service.writeToOutputStream(outputStream);
    }
}

Service Class -

@Service
public class UserService {

    @Autowired
    private UsersRepository usersRepository;

    @Transactional(readOnly = true)
    public void writeToOutputStream(final OutputStream outputStream) {
        try (Stream<UsersEntity> usersResultStream = usersRepository.findAllByCustomQueryAndStream()) {
            try (ObjectOutputStream oos = new ObjectOutputStream(outputStream)) {

                usersResultStream.forEach(emp -> {
                    try {
                        oos.write(emp.toString().getBytes());
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                });
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
}

Complete Code is available at github - https://github.com/bagesh2050/HttpResponseStreamingDemo

Still, I am willing for suggestions related to Http Streaming. Please provide if you have better ideas.

Answer 2

No sample shows "such complex" usage of StreamingResponseBody , and I fear it is "not possible" (at least I couldn't manage/fix it, with StreamingResponseBody and Stream query)

...but, what was possible:

1. Use findAll() (the normal unstreamed List-repo method) within StreamingResponseBody.

   (But I understand the "need" of doing the web request asynchronously... and the db request "streamed"...)

2. Use Callable (async web request) and an @Async CompletableFuture<..> (async db request):

    @RestController @RequestMapping("/api") public class UsersController { @Autowired private UsersRepository usersRepository; @GetMapping(value = "/async/users") public Callable<List<UsersEntity>> fetchUsersAsync() { Callable callable = () -> { return usersRepository.readAllBy().get(); }; return callable; } }

   ..and a Repository like:

    @Repository public interface UsersRepository extends JpaRepository<UsersEntity, Integer> { @Async CompletableFuture<List<UsersEntity>> readAllBy(); }

   (see spring-samples ).. don't forget to @EnableAsync on your application/configuration:

    @org.springframework.scheduling.annotation.EnableAsync @SpringBootApplication public class Application {... }

Sorry, it is not even an answer, but my findings - too long for a comment.

The asynchronous web request can be achieved in various ways. (see https://spring.io/blog/2012/05/10/spring-mvc-3-2-preview-making-a-controller-method-asynchronous/ , https://niels.nu/blog/2016/spring-async-rest.html , and even not mentioned "reactive" api)