Attach unlimited sqlite databases using php

Question

One devices generates .db files every 10 minutes with different rows. All the sqlite files have the same table called "wifi".

I'm writing a php application to read all data from all the files. I am using ATTACH method to combine databases. Like:

try {
$file_db = new PDO('sqlite::memory:');

$file_db->setAttribute(PDO::ATTR_ERRMODE,
                        PDO::ERRMODE_EXCEPTION);

$file_db->exec("ATTACH DATABASE 'wifi_16-09-02_09_44_06.db' AS db1");
//many more attached files here, like 1000 or 2000

$result = $file_db->query('SELECT * FROM db1.wifi UNION ALL SELECT * FROM db2.wifi, ...');

$file_db = null;
}
catch(PDOException $e) {
echo $e->getMessage();

Error:

SQLSTATE[HY000]: General error: 1 too many attached databases - max 10

After I wrote this code, I found out the sqlite attach has a limit, by default it is 10, and can be increased up to 125 only according to this .

How can I attach thousands of databases? if my method is wrong, what is the best way to solve this?

Answer 1

Having thousands of databases is wrong.

You should put all the data into a single table in a single database.

If that is not possible, do it anyway, by creating a temporary database and copying the data one by one.

Answer 2

There are some ways for dealing with this problem. For instance, you can change your programming approach, so you do not need to attach a lot of sqlite databases. However, for passing this limitation, I think you can change the compilation options For SQLite, for more information please take a look at following link: http://www.sqlite.org/compile.html

Answer 3

Just open each new database as a new PDO instance.

$file_db = new PDO('sqlite::wifi_16-09-02_09_44_06.db');
$file_db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$result = $file_db->query('SELECT * FROM db1.wifi UNION ALL SELECT * FROM db2.wifi, ...');
// do whatever
$file_db = new PDO('sqlite::wifi_whatever.db');
$file_db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$result = $file_db->query('SELECT * FROM db1.wifi UNION ALL SELECT * FROM 

and so on.

As simple as that.