How does MySQL handle gigantic inserts while in a transition (InnoDB)?

Question

I'm working on a project where huge imports, in my case meaning some 20 000 rows with 20 columns of data, and was planning on some home-made unsafe solution for making sure that no data would be imported if an error arose midway.

I assumed the MySQL transaction would be unfit for this purpose considering the amount of data I would be working with. Just to be sure though, I tested it out. Before the import I start a transaction, concluding the import I commit the it. At the commit statement, 20 000+ rows are smoothly inserted to the table in a matter of milliseconds without any notable memory or CPU usage during the import.

I can't wrap my mind around it! How does this work? Where is the limit? I've skimmed through the TRANSACTION documentation and can't find any information on how it operates under the bonnet. I'm thinking surely MySQL is pulling some tricks and I should be aware of in case of a backfire.

I'll accept any answer with some solid information on the above paragraph

Answer 1

The is only limited by the redo_log_size, and your hardware specs of course. If you do large transactions (20.000 is decent enough, but not a couple of million transactions is not unheard of, to put things in perspective), make sure the inno_db_log_file_size=[accepptable file size, like 256M of more] is set to a large enough number.

More useful info on the excellent mysqlperformanceblog: http://www.mysqlperformanceblog.com/2009/01/13/some-little-known-facts-about-innodb-insert-buffer/