how to properly set paginaging logic for data which come from multiple paginated api

Question

In my web api project I have /search endpoint to search academic publications based on user input. I am making search call to four different api in that search ep. That four api have their paging mechanism. I am having trouble to create paging logic for my /search ep caller.

How can I get pageSize and pageNumber as a parameter from search ep and return number of pageSize item from that 4 apis. I would get all items from that 4 apis and return number of pageSize item in each call, but I do not wanna get all items from apis in each call.

For example, I get a request like this:

/seach?pageNumber=1&pageSize=30

I will do following in search function:

var apiResult1 = GetApiOneResult(); // apiResult1 = { page : 1, pageSize: 10, total: 100, data: [...] }
var apiResult2 = GetApiTwoResult(); // apiResult2 = {page: 1, pageSize: 10, total: 50, data: [...] }
var apiResult3 = GetApiThreeResult(); // apiResult3 = {page: 1, pageSize: 10, total: 0, data: null }

In above scenario search will return 20 item as my apiResult3 have no data, but I want to show 30 data. I can call apione 2nd page and get 10 more item. If I do that when client ask for 2nd page ( /search?pageNumber=2&pageSize=30 ) I shouldn t call apiOne's second page as I already showed that in page one.

This is just one scenario, I need to think many more things probably. So how can I achieve this paging problem preferably only in my backend code?

I know this is not a specific question but any code example or maybe already created design pattern will help me.

Answer 1

Write a iterator wrapper around each source. It will make everything easy.

In a basic form, if you are getting pages from the api, you can store that in the iterator and return items one by one. If you run out of items you can request the next page for that source. The page size you use with the source matters only for performance and memory usage, and has nothing to do with the final result.

You can use Enumerable.Concat to concatenate the different iterators, to make them look like a single list.

Then you can skip to element (pageNumber - 1) * pageSize and copy pageSize results into your output. You can do this one by one since the underlying iterators will do the caching/batching for you.

To speed it up you can try to get the skip part to avoid doing a call, or jump over multiple items.

NOTE : This woks well if the data is frozen, or rarely changes. If it changes while you go to a next page you might be missing items or seeing duplicates.