distinctUntilChanged not working (e.g. different list size is not considered different)

Question

My observable looks like following:

obs
    .doOnNext { logger("Items changed (${it.size})") }
    .distinctUntilChanged()
    .doOnNext { logger("Items changed (${it.size})- EMIITTED") }

Log looks like following:

Items changed (7)
Items changed (7)- EMIITTED
Items changed (8)
// => missing EMIITTED message although it.size has changed => WHY?

Using the default comparator with a list of comparable items seems to fail here. Why? If the observables emitted list item size changed, the data is different, so distinctUntilChanged should not filter out the new list. But it seems like this happens here. Why?

Do I really need to provide my own comparator for distinctUntilChanged if I emit a list of items that compares the list size and the items one by one?

Edit

My obs basically looks like following:

obs = Observable.combineLatest(
    RxDBDataManager.appsManager.observeList(),
    RxDBDataManager.widgetsManager.observeList(),
    RxDBDataManager.shortcutsManager.observeList(),
    RxDBDataManager.customItemsManager.observeList(),
    RxDBDataManager.foldersManager.observeList(),
    Function5<List<IDBApp>, List<IDBWidget>, List<IDBShortcut>, List<IDBCustomItem>, List<IDBFolder>, List<IFolderOrSidebarItem>> { t1, t2, t3, t4, t5 ->
        val list = ArrayList<IFolderOrSidebarItem>()
        list.addAll(t1)
        list.addAll(t2)
        list.addAll(t3)
        list.addAll(t4)
        list.addAll(t5)
        list
    }
    .flatMapSingle {
        Observable.fromIterable(it)
                .filter { it.parentType == parentType && it.parentId == parentId }
                .cast(T::class.java)
                .toList()
    }
    .flatMapSingle {
        Observable.fromIterable(it)
                .sorted(comparator)
                .toList()
    }

Additionally I apply some sorting and filtering on this data with

Answer 1

Based on the exchange in the comments:

RxJava users are encouraged to use immutable data types in its flows which prevents concurrency issues such as modifying the same object at different stages from different threads resulting in broken operator behavior and seemingly impossible business logic failures.

In this case, distinctUntilChanged didn't work as expected because mutable items were changed in a way that two subsequent onNext signals basically had the same content and the operator filtered them out as being non-distinct.

A way to detect if the items involved are in fact the same unintentionally is to use the bi-predicate version of the operator and then placing a breakpoint in the custom lambda. This lets one inspect the previous and current values and see if they are truly equal even if they shouldn't be:

source.distinctUntilChanged((prev, curr) -> {
    // breakpoint the next line
    return prev.equals(curr);
});

As in this case, broken behavior was due to a mutable item changed somewhere and thus evaluating as the same as the current/previous. With List s, it is often not practical to breakpoint all mutation methods (such as add , addAll , set , remove etc.) but one can turn a mutable list into an immutable one and send it along the sequence. The built-in way is to convert it via the Collections::unmodifiableList :

  source
  .toList()
  .map(Collections::unmodifiableList)
  ;

This will crash whenever a mutation is attempted on the now unmodifiable list instance, pointing to the logic that should be investigated further.