How to use pagination using RxSwift and Alamofire?

Question

I am trying to consume an api with alamofire and rxswift. I have written the methods but the onNext of the observer is getting called only once. I am trying to do it with recursive call. What is wrong with this? Api will return 10 object at a time based on the timestamp. So I am checking if just returned array contains 10 objects. If yes then there are more, if not then that's the end.

func fetchPersonalization(fromList:[Personalization],timeStamp:Int) -> Observable<PersonalizationContainer>
    {


        let dictHeader = ["Accept":"application/json","regid" :  pushtoken , "os" : "ios" , "token" : token , "App-Version" : "1324" ,  "Content-Type" :  "application/json"]

        return fetchPersonalizationUtil(dictHeader: dictHeader, timeStamp: timeStamp)
            .flatMap { (perList) -> Observable<PersonalizationContainer> in

                    let persoList:[Personalization] = perList.list
                    let finalList = fromList + persoList
                    if(persoList.count==10){
                        let newTimeStamp = persoList.last!.lastModifiedAt! - 1

                        return Observable.merge(Observable.just(PersonalizationContainer(l: finalList, d: perList.data)),
                            self.fetchPersonalization(fromList:finalList,timeStamp: newTimeStamp)
                        )
                            //self.fetchPersonalization(fromList:finalList,timeStamp: newTimeStamp)



                    }else {
                        return Observable.just(PersonalizationContainer(l: finalList, d: Data()))
                    }

        }
    }

    func fetchPersonalizationUtil(dictHeader:[String:String],timeStamp:Int) -> Observable<PersonalizationContainer>
    {

        return Observable<PersonalizationContainer>.create({ (observer) -> Disposable in
            Alamofire.request("https://mranuran.com/api/hubs/personalization/laterthan/\(timeStamp)/limit/10/" ,headers: dictHeader).responseData { response in
                if let json = response.result.value {
                    //print("HUBs JSON: \(json)")

                    do {
                        let list = try JSONDecoder().decode([Personalization].self, from: json)
                        let pContainer = PersonalizationContainer(l: list, d: json)
                        print("ANURAN \(list[0].name)")
                        observer.onNext(pContainer)
                        observer.onCompleted()
                    }catch {
                        print(error)
                        observer.onError(error)
                    }

                }
                else{
                    observer.onError(response.result.error!)
                }
            }

            return Disposables.create()
        })

    }

I put a break point on the onNext method and it seemed it's getting called only once. Stuck with this for hours and RxSwift's GithubRepo example in their official github repo, I can't figure it out what they are doing. What can be wrong with my process?

Answer 1

I wrote this up a while back using Promises, here it is using Singles.

You pass in:

1. the seed which is used to make the first network call.
2. the pred which will be given the results of the most recent call and produces either an argument to make the next network call or nil if done. (In here is where you would check the count and return the next time stamp if another call is required.)
3. the producer which makes the network call.

It eventually returns a Single with an array of all the results. It will error if any of the internal network calls error out.

func accumulateWhile<T, U>(seed: U, pred: @escaping (T) -> U?, producer: @escaping (U) -> Single<T>) -> Single<[T]> {
    return Single.create { observer in
        var disposable = CompositeDisposable()
        var accumulator: [T] = []
        let lock = NSRecursiveLock()
        func loop(_ u: U) {
            let product = producer(u)
            let subDisposable = product.subscribe { event in
                lock.lock(); defer { lock.unlock() }
                switch event {
                case let .success(value):
                    accumulator += [value]
                    if let u = pred(value) {
                        loop(u)
                    }
                    else {
                        observer(.success(accumulator))
                    }
                case let .error(error):
                    observer(.error(error))
                }
            }
            _ = disposable.insert(subDisposable)
        }
        loop(seed)
        return disposable
    }
}

I don't think the lock is actually necessary, but I put it in just in case.

Answer 2

I've improved, based on @Daniel T.'s answer , by adding next page loading trigger. This is useful when the next page should be loaded only when user scrolls to the bottom of UITableView of in similar cases.

First page is loaded instantly upon subscribe and each subsequent page right after receiving a signal in nextPageTrigger parameter

Example usage:

let contents = loadPagesLazily(
    seed: 1,
    requestProducer: { (pageNumber: Int) -> Single<ResponseContainer<[Content]>> in
        return dataSource.loadContent(page: Id, pageSize: 20)
    },
    nextKeySelector: { (responseContainer: ResponseContainer<[Content]>) -> Meta? in
        let hasMorePages = responseContainer.meta.currentPage < responseContainer.meta.lastPage

        return hasMorePages ? responseContainer.meta.currentPage + 1 : nil
    },
    nextPageTrigger: loadMoreTrigger
)

return contents
    .scan([]], accumulator: { (accumulator, nextPageContainer) -> SearchResults in
        accumulator + nextPageContainer.data
    })

Parameters:

• seed - first page loading information PageKey
• requestProducer - transforms each PageKey to a page loading Single
• nextKeySelector - creates next page loaging info based on data retrieved in eah page resulted from requestProducer call. Return nil here if there is no next page.
• nextPageTrigger - after receiving first page each subsequent page is returned only after receiving a .next signal in this observable/

func loadPagesLazily(
    seed: PageKey,
    requestProducer: @escaping (PageKey) -> Single<Page>,
    nextKeySelector: @escaping (Page) -> PageKey?,
    nextPageTrigger: Observable<Void>
) -> Observable<Page> {
    return requestProducer(seed)
        .asObservable()
        .flatMap({ (response) -> Observable<Page>  in
            let nextPageKey = nextKeySelector(response)

            let nextPageLoader: Observable<Page> = nextPageKey
                .map { (meta) -> Observable<Page> in
                    nextPageTrigger.take(1)
                        .flatMap { (_) -> Observable<Page> in
                            loadPagesLazily(
                                seed: meta,
                                requestProducer: requestProducer,
                                nextKeySelector: nextKeySelector,
                                nextPageTrigger: nextPageTrigger
                            )
                        }
                } ?? Observable.empty()

            // Concatenate self and next page recursively
            return Observable
                .just(response)
                .concat(nextPageLoader)
        })
}