使用WebClient和Flux進行多重異步其余分頁調用

Question

我需要使用webclient和flux調用其余的分頁API。 我已經嘗試過一種阻塞的方式（一個接一個），但是我想使其平行。讓我們一次說10個並行調用。 每個調用獲取1000條記錄。
我已經在調用第0個請求以從標題中獲取總記錄數。 請求完成后，我需要調用POST api發送此響應（1000條記錄）。

如果任何請求完成，則將發送第11個請求，依此類推。 我已經看過asyncRestTemplate和可聽期貨的其他示例，但是asyncRestTemplate已經棄用，替代方法是spring-webflux

也
由於rest模板將被棄用

我做了什么。

1. 除以總數/ 1000->給出總頁數
2. 循環直到5（如果我更改為totalpages count，那么它會給我500內部服務器錯誤）
3. 調用返回Mono的服務>
4. 訂閱每個請求

ObjectMapper objmapper = new ObjectMapper();

HttpHeaders headers = partsService.getHeaders();
long totalCount = Long.parseLong(headers.get("total-count").get(0));
log.info(totalCount);
long totalPages = (long) Math.ceil((double) totalCount / 1000);
log.info(totalPages);
// List<Mono<List<Parts>>> parts = new ArrayList<>();
for (long i = 1; i <= 5; i++) {
    partsService.fetchAllParts(1000L, i).log().subscribe(partList -> {
        try {
            // post each request response to another API
            log.info(objmapper.writeValueAsString(partList));
        } catch (JsonProcessingException ex) {
            ex.printStackTrace();
        }

    });
    log.info("Page Number:" + i);
}

我想在沒有任何outOfmemoryerror情況下並行執行，並且不給調用api帶來太多負擔。 另外，我嘗試一次獲取所有頁面，但是卻收到500 Internal server error。

我是Flux（項目反應堆）的新手

實施以下解決方案

它沒有並行運行，單個請求大約需要2分鍾的時間，這意味着所有10（並發級別）都應同時完成。

try {
        fetchTotalCount().log()
                .flatMapMany(totalCount -> createPageRange(totalCount, 1000)).log()
                .flatMap(pageNumber -> fetch(1000, pageNumber), 10).log()
                .flatMap(response -> create(response))
                .subscribe();

        } catch (Exception e) {
            e.printStackTrace();
        }

日志

2019-07-29T09:00:14,477 INFO  [scheduling-1] r.u.Loggers$Slf4JLogger: request(10)
2019-07-29T09:00:14,478 INFO  [scheduling-1] r.u.Loggers$Slf4JLogger: request(10)
2019-07-29T09:00:14,479 INFO  [scheduling-1] r.u.Loggers$Slf4JLogger: request(unbounded)
2019-07-29T09:00:14,679 INFO  [scheduling-1] c.o.q.l.Logging: fetch() execution time: 546 ms
2019-07-29T09:00:17,028 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(74577)
2019-07-29T09:00:17,042 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(1)
2019-07-29T09:00:17,068 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,1) execution time: 24 ms
2019-07-29T09:00:17,078 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(2)
2019-07-29T09:00:17,080 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,2) execution time: 2 ms
2019-07-29T09:00:17,083 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(3)
2019-07-29T09:00:17,087 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,3) execution time: 2 ms
2019-07-29T09:00:17,096 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(4)
2019-07-29T09:00:17,098 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,4) execution time: 1 ms
2019-07-29T09:00:17,100 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(5)
2019-07-29T09:00:17,101 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,5) execution time: 1 ms
2019-07-29T09:00:17,103 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(6)
2019-07-29T09:00:17,106 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,6) execution time: 3 ms
2019-07-29T09:00:17,108 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(7)
2019-07-29T09:00:17,110 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,7) execution time: 2 ms
2019-07-29T09:00:17,113 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(8)
2019-07-29T09:00:17,115 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,8) execution time: 1 ms
2019-07-29T09:00:17,116 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(9)
2019-07-29T09:00:17,118 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,9) execution time: 1 ms
2019-07-29T09:00:17,119 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(10)
2019-07-29T09:00:17,121 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,10) execution time: 1 ms
2019-07-29T09:00:17,123 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onComplete()
2019-07-29T09:09:03,295 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-29T09:09:03,296 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(11)
2019-07-29T09:09:03,296 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,11) execution time: 0 ms
2019-07-29T09:09:03,730 INFO  [reactor-http-nio-1] c.o.q.s.Scheduler: 200 OK
2019-07-29T09:09:03,730 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-29T09:09:05,106 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(// data print)
2019-07-29T09:09:05,196 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-29T09:09:05,196 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(12)
2019-07-29T09:09:05,198 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,12) execution time: 1 ms
2019-07-29T09:09:05,466 INFO  [reactor-http-nio-1] c.o.q.s.Scheduler: 200 OK
2019-07-29T09:09:05,466 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-29T09:09:09,565 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(//  data print)
2019-07-29T09:09:09,730 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-29T09:09:09,730 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(13)
2019-07-29T09:09:09,731 INFO  [reactor-http-nio-1] c.o.q.l.Logging: fetch(1000,13) execution time: 0 ms
2019-07-29T09:09:10,049 INFO  [reactor-http-nio-1] c.o.q.s.Scheduler: 200 OK

更新資料

在更正調用API之后，記錄即將到來，但是在獲取最后一頁（75）之后，我得到了404 Not found錯誤。

2019-07-30T14:07:50,071 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(75)
2019-07-30T14:07:50,075 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onComplete()
2019-07-30T14:07:50,322 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(200 OK)
2019-07-30T14:07:50,323 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-30T14:07:51,973 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(//data)
2019-07-30T14:07:52,440 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(200 OK)
2019-07-30T14:07:52,440 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-30T14:07:54,522 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(//data)
2019-07-30T14:07:54,699 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(//data)
2019-07-30T14:07:55,075 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(200 OK)
2019-07-30T14:07:55,076 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-30T14:07:55,371 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onNext(200 OK)
2019-07-30T14:07:55,371 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: request(1)
2019-07-30T14:07:55,471 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: cancel()
2019-07-30T14:07:55,472 INFO  [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: cancel()
2019-07-30T14:07:55,473 ERROR [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: onError(java.lang.Exception: 4XX received from API)
2019-07-30T14:07:55,473 ERROR [reactor-http-nio-1] r.u.Loggers$Slf4JLogger: 
java.lang.Exception: 4XX received from API

Answer 1

Flux.flatMap具有用於設置並發級別的參數，可讓您協調並行化。

在下面的示例中，我使用了偽URL，示例中的一些片段以及一些其他簡單的代碼來演示如何實現此目的：

public static void main(String[] args)
{
    fetchTotalCount()
            .flatMapMany(totalCount -> createPageRange(totalCount))
            .flatMap(pageNumber -> fetch(pageNumber), 5) // 5 is the concurrency level = how many pages we query concurrently
            .flatMap(response -> process(response))
            .subscribe();
}

private static Mono<Integer> fetchTotalCount()
{
    return webClient.get()
                    .uri("http://www.example.com/get-total-count")
                    .exchange()
                    .map(ClientResponse::headers)
                    .map(headers -> headers.asHttpHeaders().get("total-count").get(0))
                    .map(Integer::valueOf);
}

private static Flux<Integer> createPageRange(int totalCount)
{
    int totalPages = (int) Math.ceil((double) totalCount / 1000);

    return Flux.range(1, totalPages);
}

private static Mono<Response> fetch(int pageNumber)
{
    return webClient.get()
                    .uri("http://www.example.com/fetch?page=" + pageNumber)
                    .retrieve()
                    .bodyToMono(Response.class);
}

private static Mono<Response> process(Response response)
{
    // todo send other http request for the post api here
    return Mono.just(response);
}

private static class Response
{
}

Answer 2

使用.flatMap()的重載版本，您可以在其中指定並發級別。 通常， .flatMap()熱切地訂閱所有內部流-在訂閱下一個流之前，它不會等待任何流完成（與.concatMap()不同）。 但是您指定了一個並發級別，它只會急切地（一次）訂閱那么多內部流。 在您的情況下，僅當最初的10個內部流中的至少一個完成時，第11個內部流才會被訂閱。 這恰好解決了您的問題。