How to integrate Redux with very large data-sets and IndexedDB

Question

I have an app that uses a sync API to get its data, and requires to store all the data locally. The data set itself is very large, and I am reluctant to store it in memory, since it can contains thousands of records. Since I don't think the actual data structure is relevant, let's assume I am building an email client that needs to be accessible offline, and that I want my storage mechanism to be IndexedDB (which is async).

I know that a simple solution would be to not have the data structure as part of my state object and only populate the state with the required data (eg - store email content on state when EMAIL_OPEN action is triggered). This is quite simple, especially with redux-thunk.

However, this would mean I need to compromise 2 things:

1. The user data is no longer part of the "application state", although in truth it is. Since the sync behavior is complex, and removing it from the app state machine will hurt the elegance of the redux concepts (the way I understand them)
2. I really like the redux architecture and would like all of my logic to go through it, not just the view state.

Are there any best-practices on how to use redux with a not-in-memory state properties? The thing I find hardest to wrap my head around is that redux relies on synchronous APIs, and so I cannot replace my state object with an async state object (unless I remove redux completely and replace it with my own, async implementation and connector).

I couldn't find an answer using Google, but if there are already good resources on the subject I would love to be pointed out as well.

UPDATE: Question was answered but wanted to give a better explantation into how I implemented it, in case someone runs into it:

The main idea is to maintain change lists of both client and server using simply redux reducers, and use a connector to listen to these change lists to update IDB, and also to update the server with client changes:

1. When client makes changes, use reducers to update client change list.
2. When server sends updates, use reducers to update server change list.
3. A connector listens to store, and on state change updates IDB. Also maintain internal list of items that were modified.
4. When updating the server, use list of modified items to pull delta from IDB and send to server.
5. When accessing the data, use normal actions to pull from IDB (eg using redux-thunk)

The only caveat with this approach is that since the real state is stored in IDB, so we do lose some of the value of having one state object (and also harder to rewind/fast-forward state)

Answer 1

I think your first hunch is correct. If(!) you can't store everything in the store, you have to store less in the store. But I believe I can make that solution sound much better:

IndexedDB just becomes another endpoint, much like any server API you consume. When you fetch data from the server, you forward it to IndexedDB, from where your store is then populated. The store gets just what it needs and caches it as long as it doesn't get too big or stale.

It's really not different than, say, Facebook consuming their API. There's never all the data for a user in the store. References are implemented with IDs and these are loaded when required.

You can keep all your logic in redux. Just create actions as usual for user actions and data changes, get the data you need and process it. The interface is still completely defined by the user data because you always have the information in the store that is needed to GET TO the rest of it when needed. It's just somewhat condensed, ie you only save the total number of messages or the IDs of a mailbox until the user navigates to it.