How to improve insert performance on a billion row table?

Question

I have billion row table that no longer fits in the memory.

When I insert new rows in bulk, the overhead of recounting the primary index, kills the performance. I HAVE to have this index because otherwise SELECT statements are really slow. But since the inserts come in a random order, with each row inserted, the data has to be written in different area of the disk.

And since the HDD is capped at 200 IO operations per second, this slows the inserting to a crawl.

Can I "have my cake and eat it" at the same time in this situation? Maybe by creating another table in which the data would be grouped by different column ( by having a different primary key )? But this seems wasteful to me and I don't even know if that would help...

Or maybe I could use some staging table? Insert there 1,000,000 rows and then insert them to the target table, grouped up by the primary key?

Am I doomed?

EDIT:

I've partitioned the table horizontally.

When I removed the primary key on this field that I need and placed it on the autoincrement field, the inserts were blazingly fast.

Unfortunately, since the data on disk is placed by the primary key value, this killed the select performance... because selects don't query based on the autoincrement value but rather on the PK value.

So either I insert rows fast or I select them fast. Isn't there any solution that could help in both cases?

Answer 1

.When you insert new row each time it will do indexing after data insert. It's take more time. You can use

START TRANSACTION

...You r insert query...

COMMIT

Answer 2

Try Like this

mysql_query("START TRANSACTION");

your insert query

mysql_query("COMMIT");