Data sync from MySQL to NoSQL key-value store

Question

I am having a legacy system with the MySQL at the backend and python as the primary programming language.

Recently we have a scenario where we need to display a dashboard with the information present in the MySQL database. The data in the table changes every second.

This can be thought of similar to a bid application where people bid constantly. Every time a user bids a record goes in to the database. When an user updates his bid it updates the previous value.

I also have few clients who monitor this dashboard which updates the statistics.

• I need to order this data in realtime as people bid in real time.
• I don't prefer to run queries against MySQL because at any second I may have few 1000 clients querying the database. This will create load on database.

Please advice.

Answer 1

If you need to collect and order data in realtime you should be looking at the atomic ordered map and ordered list operations in Aerospike.

I have examples of using KV-ordered maps at rbotzer/aerospike-cdt-examples .

You could use a similar approach with the user's ID being the key, the bid being a list with the structure [1343, { foo: bar, ts: 1234, this: that} ] . The bid amount in cents (integer) is the first element of the list, all the other information is in a map in the second element position.

This will allow you to update a user bid with a single map operation, get back the user's bid with a single operation, order by rank (on the bid amount) to get the top bids ordered, get all the bids in a particular range, etc. You would have one record per item, with all the bids in this KV-sorted map.