how to handle very large amount of data in running project?

Question

I have a project in rails in which we have table product which stores very large amount of records and data grows everyday when we do query on that table it takes almost 2 to 4 minutes.

My question is how I can handle it project is already built it is no longer in state where we can redesign its architecture?

I guess this is problem of handling big data but I am not aware of standards or practice to do that.

I tried to apply Indexing it helps but still data is too much I need another technique to handle it may be I need to split it into two tables but not sure.

the goal is to keep storing incoming data daily and do queries in milliseconds not in minutes.

Answer 1

If you're committed to postgresql tables, as it seems you are, here's a course of action you should take.

Start by reading this. https://stackoverflow.com/tags/query-optimization/info

Identify the three queries in your application that take the most elapsed time. Those could be long-running report-generation queries, or frequently-run transactional queries.

Use postgresql's EXPLAIN ANALYZE command to figure out how your server satisfies them. You'll have to learn a lot the first few times you gaze upon this SQL Execution Plan stuff. But it is worth your time to learn.

Come back here and ask query-optimization questions as you need to.

Add or adjust table indexes.

Repeat these steps for the next three slowest queries.

Plan on doing this every few months as a routine maintenance operation as long as your app continues to grow (hopefully for many years ).

Investigate your slowest queries before you commit to restructuring your data. For example, splitting your data into tables by month, or whatever, will be a lot of work and will add testing and ops complexity. Be sure that is justified before doing it, and that it will actually help. Many properly indexed SQL tables scale up to hundreds of megarows very well indeed.