pgsql查询“ SELECT .. IN”的局限性

Question

I have a .csv file with about 14000 objectIDs. My goal is to retrieve certain fields associated with these objectIDs. So far what I have done is concatenate all the objectIDs into a comma separated list that I append to the end of this query:

SELECT objectName, objectType FROM objectTable WHERE objectID IN 1001, 1002, 1003... 

This however is very (very) slow as my database has about 16 million unique objectIDs. Is there a better way to structure such a query? Must I run this in batches? (I tried this too but it was unbearably slow) or is my entire approach wrong?

Answer 1

Load the objectIDs into a table and then join against that.

SELECT objectName, objectType
FROM objectTable INNER JOIN objectids ON (objecttable.objectid = objectids.id)

Answer 2

w/ 16m objectIDs, it's probably just taking a lot longer to upload the query string than to actually run it.

Create a table from your CSV file, with all the objectIDs preloaded. Say you call this table "objectIDs", and the main row is "id". Now you can say:

SELECT objectName, objectType FROM objectTable
INNER JOIN objectIDs ON objectIDs.objectID=objectTable.objectID

The inner join will automatically cull out any unjoined things in objectTable, and will join in a 1:1 relationship with your IDs table.

Answer 3

If you already have a comma-separated string containing all numbers, you could use a prepared statement - with the syntax of whatever client you use. Example in plain SQL:

PREPARE myplan (text) AS
    SELECT o.objectname, o.objecttype
    FROM   (SELECT unnest(string_to_array($1, ','))::int AS objectid) x
    JOIN   objecttable o USING (objectid);

EXECUTE myplan('1001, 1002, 1003');

Or , if you start from a valid CSV file on the database server , create a temporary table, COPY the data to it ( COPY is very fast), and then JOIN to it.

CREATE TEMP TABLE tmp_x (objectid int);

COPY tmp_x FROM '/path/to/my/file.csv';

    SELECT o.objectname, o.objecttype
    FROM   tmp_x
    JOIN   objecttable o USING (objectid);

DROP TABLE tmp_x;   -- optional; dropped automatically at end of session

If your file is on a different machine, use psql 's meta-command \\copy instead.

You surely have an index on objecttable.objectid ? That's crucial.