RxJava – how to wait for subscriber to process stream

Question

This is exactly the same problem as this:

RX: How to wait for subscribers to finish?

I have a producer which can produce object faster than subscriber so I want it to produce objects in parallel but block after some fixed number of items.

Like this:

1. publisher produces items
2. as soon as 10 items are produced, subscriber takes them and start processing
3. publisher continues to produce and buffer items in parallel, but if next 10 are produced, then it waits for subscriber
4. after finishing processing, subscriber immediately takes next 10 objects from the buffer
5. publisher continues to produce and buffer items
6. etc.

Current solution:

            val bars = Flowable.fromStream(foos)
// calculate is faster
                    .map(foo -> calculateBars(foo))
                    .buffer(10)
                    .observeOn(Schedulers.single())
// than saving to database
                    .flatMap(bars -> Flowable.fromStream(saveBars(bars)))
                    .collect(Collectors.toSet())
                    .blockingGet();

If I add .observeOn(Schedulers.single()) then publisher doesn't wait at all for consumer and eventually crash with OutOfMemoryError. If I remove it it waits all the time instead of preparing the next batch of items.

This is related to my other question Keep memory usage constant with JpaRepository andRxJava . I think that solving this issue will solve that.

Answer 1

After hours of trying to implement this in java, I gave up and solved the problem with kotlin coroutines

    @Transactional(readOnly = true)
    open fun calculate(): Set<UUID> {
        val fooIds = findOperators()
        val saved = mutableListOf<UUID>()
        runBlocking(Dispatchers.IO) {
            // Run calculations in a separated thread, which communicates
            // with the main thread via a channel of batches.
            // Meanwhile, the main thread is busy saving records to database as soon as they arrive
            // from the calculation thread.
            val channel = Channel<Collection<Bar>>(1)
            launch {
                var batch = mutableListOf<Bar>()
                for (id in fooIds) {
                    val records = findRecordsByFoo(id)

                    if (batch.size == BATCH_SIZE) {
                        channel.send(batch)
                        batch = mutableListOf()
                    }
                }
                if (batch.isNotEmpty()) {
                    channel.send(batch)
                }
                try {
                    channel.close()
                }
                catch (e: ClosedReceiveChannelException) {
                    log.info("Finished processing foos")
                }
            }
            for (bars in channel) {
                val uuids = saveBars(bars)
                saved.addAll(uuids.asSequence())
            }
        }
        fooIds.close()
        return saved.toSet()
    }

Edit: These a required gradle changes:

plugins {
    id 'org.jetbrains.kotlin.jvm' version '1.7.10'
}
dependencies {
    implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4'
}
// to put the kotlin file close to a java service which calls it:
sourceSets {
    main.kotlin.srcDirs += 'src/main/java'
    main.java.srcDirs += 'src/main/java'
}