Django cache real-time data with DRF filtering and sorting

Question

I'm building a web app to manage a fleet of moving vehicles. Each vehicle has a set of fixed data (like their license plate), and another set of data that gets updated 3 times a second via websocket (their state and GNSS coordinates). This is real-time data.

In the frontend, I need a view with a grid showing all the vehicles. The grid must display both the fixed data and the latest known values of the real-time data for each one. The user must be able to sort or filter by any column.

My current stack is Django + Django Rest Framework + Django Channels + PostgreSQL for the backend, and react-admin (React+Redux+FinalForm) for the frontend.

The problem: Storing real-time data in the database would easily fulfill all my requirements as I would benefit from built-in DRF and Django's ORM sorting/filtering, but I believe that hitting the DB 3 times/sec for each vehicle could easily become a bottleneck when scaling up.

My current solution involves having the RT data in python objects that I serialize/deserialize (pickle) into/from the Django cache (REDIS-backed) instead of storing them as models in the database. However, I have to manually retrieve the data and DRF-serialize it in my DRF views. Therefore, sorting and filtering don't work as there are no SQL queries involved.

If pgsql had some sort of memory-backed tables with zero disk access it would be great, but, according to my research there is no such feature as of today.

So my question is: What would be the correct/usual approach to my problem?

Answer 1

I think you should separate your service code into smaller service:

• Receive data from vehicles
• Process data
• Separate vehicles into smaller group, use unique socket for each group

Try to update latest values as batch.

Use RAID for your database.

And I think that using cache for realtime data is wasted server resources.