CodeIgniter large batch insert

Question

I'm currently creating a batch script that calls multiple PHP scripts (that use CodeIgniter) to pull data from one DB, work with the results and insert the result in another DB. To begin, I know this might not be the best tool for the job, but it'll have to do right now. But back on topic, the script is running fine, and performance is pretty good, except for one method. It pulls data from the first DB, and inserts it into the second DB. It's a table that has about 20 columns and 35000 rows.

I'm using a simple $this->db->insert_batch('tablename', $insertdata); function to insert those results, but somehow the script takes about 15 minutes to execute...

Question is, how to optimize this process?

Thanks in advance

EDIT

Here's the query I use to create the table on the second DB, the DB that the data gets inserted into.

CREATE TABLE IF NOT EXISTS `invoices` (
    `invoice_number` varchar(40) NOT NULL,
    `shippinglist_number` varchar(40) DEFAULT NULL,
    `shippinglist_line` varchar(255) DEFAULT NULL,
    `customer_id` varchar(5) NOT NULL,
    `deptor` varchar(5) NOT NULL,
    `vat_number` int(255) DEFAULT NULL,
    `invoice_date` date NOT NULL,
    `expire_date` date NULL DEFAULT NULL,
    `currency_code` varchar(10) NOT NULL,
    `subtotal` decimal(19,4) NOT NULL,
    `vat` decimal(19,4) NOT NULL,
    `total` decimal(19,4) NOT NULL,
    `qty` int(10) NOT NULL,
    `partcode` varchar(255) NOT NULL,
    `description` text,
    `price` decimal(19,4) NOT NULL,
    `pieces_per` int(10) NOT NULL,
    `article_customer` varchar(255) DEFAULT NULL,
    `reference_customer` varchar(255) DEFAULT NULL,
    `sales_line_1` text,
    `sales_line_2` text,
    `sales_line_3` text,
    `memo` text,
    KEY `invoice_number` (`invoice_number`,`customer_id`,`deptor`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

And the query is just a normal insert, but the 35000 rows are chopped into 100 row sets.

Answer 1

Try this

// some $datas

$this->db->trans_start();

$_datas = array_chunk($datas, 300);

foreach ($_datas as $key => $data) {
    $this->db->insert_batch('table', $data);
}

$this->db->trans_complete();

you split array. 300, 300, 300 .... (mysql recommend to insert tiny data). and insert_batch! and each 300 data keep transaction. that is all. Sorry, I am not good at english.

Answer 2

Use transactions - 1 commit is always faster than 350 commits.

Edit:

Basically, you need to do the following:

// Fetch (and transform, I suppose) data from Sybase
// ...

$this->db->trans_start();

// Put your $this->db->insert_batch() calls here

$this->db->trans_commit();

Answer 3

Try using

$x = $this->db->num_rows();
for($i=0;$i<$x;$i=$i+30)
    {
    $this->db//query stuff
    $this->db->limit($i);
    $other->db->//insert stuff
    }

That way you do it line by line and you are not dependant on one huge sql query. That way you could restart whenver you wanted if some error or timeout had occurred.

Are the databases on the same server? Then you could just use a query to copy the data with a select from other database(if the same user has access to both)

Answer 4

This is an old post, however for those who landed here while searching for an answer to this may be this link may help you.

$this->db->save_queries = FALSE;

http://greeneggmedia.com/blog/entry/undocumented-codeigniter

In the background CI try to save queries for profiling, if you disable this feature you can gain lot of time.

If you dont like this answer please dont downvote. was just trying to help.