在非常大的数据库中查询字符串的最佳方法？

Question

In a Postgres database I'm running a query that searches for a string that is a sha256 hash in a table with approx. 5*10^8 rows. This query can take up to 9 seconds, which sounds ok for a single datapoint, however I need to run this join query for 10^9 times (for every item in another table). The column that contains sha256 hash is indexed, also I don't have any additional information (id or timestamp) that I could use to search for just a part of the string and that id.

My current setup is to call this slow query from a python daemon (using psycopg2), send it the id from the 10^9 rows table, and print out the execution time every 100 execution. I tried commiting every few queries, didn't make a measurable difference, autocommit is default=off.

Am I missing something that could run this query faster, or is a better choice to dump my db into something like elasticsearch, then do this searching for strings using elasticsearch?

EDIT: Explain of the slow query:

EXPLAIN UPDATE txout
SET fk_tx_id = txid.tx_id
FROM
(
 SELECT tx.tx_id, txout.tx_hash
 FROM tx tx
 INNER JOIN txout
 ON tx.tx_hash = txout.tx_hash
 WHERE txout.fk_block_id = 398361
) AS txid
WHERE txout.tx_hash = txid.tx_hash
AND txout.fk_block_id = 398361;
-[ RECORD 1 ]--
QUERY PLAN | Update on txout  (cost=149874.29..323547.14 rows=5 width=345)
-[ RECORD 2 ]--
QUERY PLAN |   ->  Nested Loop  (cost=149874.29..323547.14 rows=5 width=345)
-[ RECORD 3 ]--
QUERY PLAN |         ->  Merge Join  (cost=149873.60..150727.71 rows=19864 width=400)
-[ RECORD 4 ]--
QUERY PLAN |               Merge Cond: (txout.tx_hash = txout_1.tx_hash)
-[ RECORD 5 ]--
QUERY PLAN |               ->  Sort  (cost=77894.30..78025.39 rows=52438 width=329)
-[ RECORD 6 ]--
QUERY PLAN |                     Sort Key: txout.tx_hash
-[ RECORD 7 ]--
QUERY PLAN |                     ->  Index Scan using idx_txout_fk_block_id on txout  (cost=0.58..65716.10 rows=52438 width=329)
-[ RECORD 8 ]--
QUERY PLAN |                           Index Cond: (fk_block_id = 398361)
-[ RECORD 9 ]--
QUERY PLAN |               ->  Materialize  (cost=71979.30..72241.49 rows=52438 width=71)
-[ RECORD 10 ]--
QUERY PLAN |                     ->  Sort  (cost=71979.30..72110.39 rows=52438 width=71)
-[ RECORD 11 ]--
QUERY PLAN |                           Sort Key: txout_1.tx_hash
-[ RECORD 12 ]--
QUERY PLAN |                           ->  Index Scan using idx_txout_fk_block_id on txout txout_1  (cost=0.58..65716.10 rows=52438 width=71)
-[ RECORD 13 ]--
QUERY PLAN |                                 Index Cond: (fk_block_id = 398361)
-[ RECORD 14 ]--
QUERY PLAN |         ->  Index Scan using idx_tx_hash on tx  (cost=0.70..8.69 rows=1 width=75)
-[ RECORD 15 ]--
QUERY PLAN |               Index Cond: (tx_hash = txout_1.tx_hash)

Answer 1

It seems you're trying to setup a foreign key from one table to another over a string field. Am I correct?

Postgresql solution If that's the case, building an explicit foreign key (and the associated index) in postgresql would prove the first solution to try although it is sure that with hundreds of millions of row in one side and billions in the other, you'll need a fairly strong setup underlying your postgresql database to build the index. Once done, querying should nevertheless be reasonable.

elasticsearch solution To answer your more global question, using something like elasticsearch changes completely the problem because it uses inverted indexes to query string super efficiently, and is based on a distributed system where data is sharded on several nodes (ie several machines). Therefore, provided you have many instances in your elasticsearch cluster, you can speedup significantly text-searches by breaking down the search among the different shards (which parallelizes the search), and using the pre-computed inverted index. Neverthless, setting up an elasticsearch cluster is a commitment and the ingestion / indexing of billions of records won't be fast either.

Divide and conquer Another direction you can go is to perform the join locally on your computer, possibly splitting the full tables based on the first character of your hashes so you can "paralellize" your join with one job per first character. Also, sorting and pre-indexing both table, in postgresql and in memory can speedup such joins significantly.

It is hard providing more guidance without more details of what you're trying to do.

Answer 2

You have a 3 way join, and it not clear what that is supposed to accomplish. Why not just:

EXPLAIN (ANALYZE, BUFFERS) UPDATE txout
SET fk_tx_id = tx.tx_id
FROM
tx tx
WHERE txout.fk_block_id = 398361
and txout.tx_hash = txid.tx_hash

Also, there is not much point in executing it 5*10^8 times if there are not that many different values of fk_block_id. You will just be updating the same rows over and over and setting them to the same thing.

Answer 3

I think your query can be simplified to:

UPDATE txout
    SET fk_tx_id = tx.tx_id
    FROM tx
    WHERE tx.tx_hash = txout.tx_hash AND
          txout.fk_block_id = 398361;

For this query, you want indexes on txout(fk_block_id, tx_hash) and tx(tx_hash) .