Ending an “infinite” stream when certain conditions are met

Question

I'm trying to pull data from a REST-style web-service which delivers content in pages.

The only way I can know that I've reached the end is when I ask for a page and there are no results. I'd like to terminate the stream at that time.

I've written the following Java code. The first function pulls a single page from the web-service and returns it as a stream. The second function flatmaps the streams together into a single stream.

public Stream<ApplicationResponse> getApplications(String token, RestTemplate rt, Integer page, Integer pageSize) {
    HttpEntity<String> entity = new HttpEntity<>("parameters", getHeaders(token));
    String url = String.format("%s?PageIndex=%s&PageSize=%s", endpoint, page, pageSize);
    ResponseEntity<ApplicationCollection> ar = rt.exchange(url, HttpMethod.GET, entity, ApplicationCollection.class);
    ApplicationResponse[] res = Objects.requireNonNull(ar.getBody()).getData();

    // Do something here when res is empty, so that the stream ends

    return Arrays.stream(res);
}

public Stream<ApplicationResponse> getApplications(String token, RestTemplate rt) {
    // This function does the right thing, exept when we run out of data!
    return IntStream.iterate(1, i -> i + 1).mapToObj(i -> getApplications(token, rt, i, 500)).flatMap(Function.identity());
}

The problem is, how do I allow this to end?

If I were writing this in Python I'd raise a StopIteration exception at the point where I know there's nothing left to put onto the stream. Is there something similar I can do?

The best thing I could think of was use a null, or raise an exception to signify the end of data and then wrap up the stream into an Iterator that knows to stop when that signal is received. But is there anything more idiomatic that I can do?

Answer 1

After comments from Holger, I gave it a shot and tried Spliterator instead of Iterator . It is indeed simpler, as next and hasNext are... kinda combined into tryAdvance ? It is even short enough to just inline it into a util method, imo.

public static Stream<ApplicationResponse> getApplications(String token, RestTemplate rt)
{
    return StreamSupport.stream(new AbstractSpliterator<ApplicationResponse[]>(Long.MAX_VALUE,
                                                                               Spliterator.ORDERED
                                                                                               | Spliterator.IMMUTABLE)
    {
        private int page = 1;

        @Override
        public boolean tryAdvance(Consumer<? super ApplicationResponse[]> action)
        {
            HttpEntity<String> entity = new HttpEntity<>("parameters", getHeaders(token));
            String url = String.format("%s?PageIndex=%s&PageSize=%s", endpoint, page, 500);
            ResponseEntity<ApplicationCollection> ar = rt.exchange(url, HttpMethod.GET, entity,
                                                                   ApplicationCollection.class);
            ApplicationResponse[] res = Objects.requireNonNull(ar.getBody()).getData();

            if (res.length == 0)
                return false;

            page++;
            action.accept(res);
            return true;
        }
    }, false).flatMap(Arrays::stream);
}

---

You could implement an Iterator and create a Stream of it:

public class ResponseIterator
    implements Iterator<Stream<ApplicationResponse>>
{
    private int                   page = 1;
    private String                token;
    private RestTemplate          rt;

    private ApplicationResponse[] next;

    private ResponseIterator(String token, RestTemplate rt)
    {
        this.token = token;
        this.rt = rt;
    }

    public static Stream<ApplicationResponse> getApplications(String token, RestTemplate rt)
    {
        Iterable<Stream<ApplicationResponse>> iterable = () -> new ResponseIterator(token, rt);
        return StreamSupport.stream(iterable.spliterator(), false).flatMap(Function.identity());
    }

    @Override
    public boolean hasNext()
    {
        if (next == null)
        {
            next = getNext();
        }
        return next.length != 0;
    }

    @Override
    public Stream<ApplicationResponse> next()
    {
        if (next == null)
        {
            next = getNext();
        }
        Stream<ApplicationResponse> nextStream = Arrays.stream(next);
        next = getNext();
        return nextStream;
    }

    private ApplicationResponse[] getNext()
    {
        HttpEntity<String> entity = new HttpEntity<>("parameters", getHeaders(token));
        String url = String.format("%s?PageIndex=%s&PageSize=%s", endpoint, page, 500);
        ResponseEntity<ApplicationCollection> ar = rt.exchange(url, HttpMethod.GET, entity,
                                                               ApplicationCollection.class);
        ApplicationResponse[] res = Objects.requireNonNull(ar.getBody()).getData();
        page++;
        return res;
    }
}

It will check whether the next response is empty in hasNext() , stopping the stream. Otherwise, it will stream and flatMap that response. I have hardwired pageSize , but you can easily make that a third input for the factory method ResponseIterator.getApplications() .