How do you deal with eventual inconsistency when using Amazon RDS with Read Replica?

Question

Consider user cart and checkout: a customer can perform addItemToCart action which will be handled by main DB instance. However, getUserCartItems action might be performed on Read Replica and it might not contain result of the first action yet due to Replica Lag. Even if we try to minimize this lag, still it's possible to hit this case, so I'm wondering what solutions have you tried in production?

According to @Henrik answer , we have 3 options:

1. Wait at user till consistent.

This means we need to perform polling (regular or long polling) on the client and wait until Replica will receive update. However, I assume Replica Lag shouldn't be longer than 1-5 secs. Also, the less Replica Lag, the more performance down we will have.

2. Ensure consistency through 2PC. 

If I understood correctly, we need to combine both addItemToCart insert and getUserCartItems select into one aggregate operation on backend and return getUserCartItems as addItemToCart response. However, the next request might still not get updated info due to lag… Yes it returns immediate confirmation about successful operation and the application can continue, however proceeding to checkout requires user cart items in order to show price correctly, so we are not fixing the problem anyway.

3. Fool the client.

Application stores/caches all successfully send data and uses it for showing. Yes, this is a solution, but it definitely requires additional business logic to be implemented:

Perform getUserCartItems request;

if (getUserCartItems returned success)
  Store addItemToCart in local storage; 
else 
  Show error and retry;

Perform getUserCartItems request;

if (getUserCartItems contains addItemToCart ID)
  Update local storage / cache and proceed with it. 
else       
  Use existing data from local storage; 

How do you deal with eventual inconsistency?

Answer 1

The correct answer is to NOT send SELECT queries to a read slave if the data needs to be immediately available.

You should structure your application such that all real-time requests hit your master, and all other requests hit one of your read slaves.

For things where you don't need real-time results, you can fool the user quite well using something like AJAX requests or websockets (websockets is going to make your application a lot more resource friendly as you won't be hammering your backend servers with multiple AJAX requests).