TCP client data requesting

Question

I am trying to find an efficient way to transfer data between the server and the client without consistency errors while keeping a low network utilization. I have come against this situation many times before and have always ended up using one of the following two plans:

Plan #1: Client requests all available data (Either on connection or on first demand). Client caches the data and uses it as its own local database (Stored in RAM). Upon any changes (Addition, deletion, update) of the data, the server notifies the client which then updates its cache

Pros: Low network utilization after initial transfer (All data are cached). Dynamic update of data

Cons: Server might be required to send a huge amount of data. This could cause a network congestion or even cause the client to crash

Plan #2: Client requests some of the data on demand (eg Retrieve first 25 items of a list). If more data are required, client requests the next 25 items and so on.

Pros: Simple. No need to transfer all of the data at once

Cons: Server cannot dynamically update the data of clients since it doesn't know what data each client has. For the same reason, clients can't cache any data since they may cache an outdated item. High network utilization throughout the usage of the system.

What is the best way to deal with this?

Answer 1

If I understand correctly, you have some kind of sequential list of data, " (eg Retrieve first 25 items of a list) ".

If that's the case, you can have the server keep an "up till record x" indicator on a per connection basis. For example, say initially you send the first 25 items. The server keeps a record, client x has gotten up to record 25. The client then needs record 57, so the server send everything up till and including record 57. Let's say record 36 gets updated. The server can send that update only to the clients that have gotten up till at least record 36.

This allows caching and updates, if in fact your data is a set of sequential records.