如何在集成中使用执行器进行并行处理

Question

I want to read files from FTP and process them. I use 'executorchannel', but 'executorchannel' won't stop receiving. However, when the processing speed can't keep up, FTP keeps reading. When the files are read too much, an 'over head' exception appears. How to solve this problem

public IntegrationFlow processFileFlow() {
        //dsl
        return IntegrationFlows
                .from("ftpFileChannel")
                .enrichHeaders(h -> h.header(MessageHeaders.ERROR_CHANNEL, "exceptionChannel", true))
                .transform(transformer())
                .transform(new MyTransformer(zipProperties))
                .split()
                //.channel(MessageChannels.executor(Executors.newFixedThreadPool(3)))
                .route("headers['type']",
                        mapping -> mapping.async(true)
                                .resolutionRequired(false)
                                .ignoreSendFailures(true)
                                .subFlowMapping(DataTypeEnum.DELIVERY_DATA.getCode(),
                                        sf -> sf.channel(MessageChannels.executor(Executors.newFixedThreadPool(2,Executors.privilegedThreadFactory())))
                                                .publishSubscribeChannel(
                                                        c -> c.subscribe(s -> s.handle("fileProcess", "processDeliveryData")))
                                                .bridge())
                                .subFlowMapping(DataTypeEnum.PLACE_DATA.getCode(),
                                        sf -> sf.channel(c -> c.queue(5))
                                                .publishSubscribeChannel(
                                                        c -> c.subscribe(s -> s.handle("fileProcess", "processPlaceData")))
                                                .bridge()
                                ))
                .aggregate(t -> t
                        //t.correlationStrategy(c -> c.getHeaders().get("file_remoteFile"))
                        .outputProcessor(o -> new EventDto(o.getMessages().stream().map(m -> ((EncryptDataDto) m.getPayload()).getEventEntity()).collect(Collectors.toList())))
                )
                .handle("fileProcess", "endHandler", e -> e.advice(after()))
                .get();
    }

Answer 1

Spring Integration provides a CallerBlocksPolicy rejected execution handler.

Use a task executor with a fixed thread pool and a bounded queue, with this policy to control the concurrency. When the queue is full, the poller thread will block until space is made available by a task completing.