使用posgres和php从数据库中的一个表中获取不在另一个数据库中另一个表中的所有ID

Question

I'm using PostgreSQL 8.1, and I have two databases with one table names clients that are identical and contain +-50k rows each.

I need to obtain all ids in one table that aren't in the other, I have the following solution

$sql = "SELECT id FROM clients WHERE id NOT IN(".pg_query($conn1,'SELECT id FROM clients').")";
$result = pg_query($conn2,$sql);

Before I run this, is this a good way to do it via PHP or are the other better faster ways?

Unfortunately I cant use dblink since I do not have privileges to install it.

Answer 1

dblink

You can make use of dblink to refer to tables from another database in your query.

Then your query could look like this (one of multiple different approaches):

SELECT id
FROM   clients c
LEFT   JOIN (
    SELECT *
    FROM   dblink('dbname=mydb', 'SELECT id FROM clients')
    AS     c(id int)
    )  x USING (id)
WHERE  x.id IS NULL;

Since PostgreSQL 9.1 installing an additional module has become even simpler:

CREATE EXTENSION dblink;

Run once per database. As for the necessary privileges :

Loading an extension requires the same privileges that would be required to create its component objects. For most extensions this means superuser or database owner privileges are needed.

COPY

If dblink (or similar x-db tools are not available for you, try COPY TO / FROM to transfer contents of the foreign table:

In foreign db:

COPY clients TO '/path/to/file';

In home db:

CREATE TEMP TABLE c_tmp (LIKE clients);
COPY c_tmp FROM '/path/to/file';

SELECT id
FROM   clients c
LEFT   JOIN c_tmp x USING (id)
WHERE  x.id IS NULL;

Temp. table is dropped automatically at end of session.

Answer 2

If you can't use a database link I think you will have to do it in the middleware - and this middleware should have a lot of RAM.

I would also recommend not to use PHP for this purpose, but if you have no other possibility, I would do it like this:

$ids = array();
$idsNotExist = array();
$resultDb1 = pg_query('SELECT id FROM db1');
while ( $row = pg_fetch_dontknowtheexactfunction_row($resultDb1) ) {
  array_push($ids, $row['id']);
}

$resultDb2 = pg_query('SELECT id FROM db2');
while ( $row = pg_fetch($resultDb2) ) {
  if ( !in_array($ids, $row['id']) ) {
    array_push($idsNotExist, $row['id']);
  }
}

Best way would of course be, because ID is primary key, to write an own in_array-function which removes the ID from haystack when it's found because it will not be searched for again and the next search loop will be a bit faster.

Answer 3

SELECT id
FROM db.clients
WHERE id NOT IN (
   SELECT id
   FROM otherdb.clients
)

one single query. There's no need for multiple db connections, unless the user ID you used for the first connection doesn't have access rights on one of those tables. SQL allows fully absolute db.table.field type specifications to allow cross-db queries.