How can I improve MySQL database performance?

Question

So i have database in project Mysql .

I have a main table that have main staff for updating and inserting .

I have huge data traffic on the data . what i am doing mainly reading .csv file and inserting to table .

Everything works file for 3 days but when table record goes above 20 million the database start responding slow , and in 60 million more slow.

What i have done ?

I have applied index in the record where i think i need of it . (where clause field for fast searching) .

I think query optimisation can not be issue because database working fine for 3 days and when data filled in table it get slow . and as i reach 60 million it work more slow .

Can you provide me the approach how can i handle this ?

What should i do ? Should i shift data in every 3 days or what ? What you have done in such situation .

Answer 1

The purpose of database is to store a huge information. I think the problem is not in your database, it should be poor query, joins, Database buffer, index and cache. These are the following reason which makes your response to slow up. For more info check this link

Answer 2

Whick operation do you want to speed up?

insert operation

A good way to speed it up is to insert records in batch. For example, insert 1000 records in each insert statement:

insert into test values (value_list),(value_list)...(value_list);

other operations

If your table got tens of millions of records, everything will be slowing down. This is quite common. To speed it up in this situation, here is some advice:

• Optimize your table definition. It depends on your particular case. Creating indexes is a common way.
• Optimize your SQL statements. Apparently a good SQL statement will run much faster, and a bad SQL statement might be a performance killer.
• Data migration. If only part of your data is used frequently, you can shift the infrequently-used data to another big table.
• Sharding. This is a more complicated way, but usually used in big data system.

Answer 3

I have applied index in the record where i think i need of it

Yes, index improve the performance of SELECT query, but at the same time it will degrade your DML operation and index has to be restructure whenever you perform any changes to indexed column.

Now, this is totally depending on your business need, whether you need index or not, whether you can compromise SELECT or DML .

Currently, many industries uses two different schemas OLAP for reporting and analytics and OLTP to store real-time data (including some real-time reporting).

Answer 4

First of all it could be helpful for us to now which kind of data you want to store.

Normally it makes no sense to store such a huge amount of data in 3 days because no one ever will be able to use this in an effective way. So it is better to reduce the data before storing in the database.

eg

If you get measuring values from a device which gives you one value a millisecond, you should think if any user is ever asking for a special value at a special millisecond or if it not makes more sense to calculate the average value of once a second, minute or hour or perhaps once a day? If you really need the milliseconds but only if the user takes a deeper look, you can create a table from the main table with only the average values of an hour or day or whatever and work with that table. Only if the user goes in ths "milliseconds" view you use the main table and have to live with the more bad performance.

This all is of course only possible if the database data is read only. If the data in the database is changed from the application (and not only appended by the CSV import) then using more then one table will be error prone.

Answer 5

For the .csv file, use LOAD DATA INFILE ...

Are you using InnoDB? How much RAM do you have? What is the value of innodb_buffer_pool_size ? That may not be set right -- based on queries slowing down as the data increases.

Let's see a slow query. And SHOW CREATE TABLE . Often a 'composite' index is needed. Or reformulation of the SELECT .