无法通过索引提高 SQL 连接速度

Question

I'm totally new to SQL and I am trying to speed up join queries for very large data. I started adding indexes (but to be honest, I don't have a deep understanding of them) and not seeing much change, I decided to benchmark on a more simple, simulated example. I'm using the psql interface of PostgreSQL 11.5 on MacOS 10.14.6. The data server is hosted locally on my computer. I apologize for any lack of relevant information, first time posting about SQL.

Databases' Structures

I created two initially identical databases, db and db_idx. I never put any index or key on tables in db, while I try putting indexes and keys on tables in db_idx. I then run simple join queries within db and db_idx separately and I compare the performance. Specifically, db_idx is made of two tables:

• A client table with with 100,000 rows and the following structure:

                   Table "public.client"
       Column    |  Type   | Collation | Nullable | Default
    -------------+---------+-----------+----------+---------
     client_id   | integer |           | not null |
     client_name | text    |           |          |
    Indexes:
        "pkey_c" PRIMARY KEY, btree (client_id)

• A client_additional table with 70,000 rows and the following structure:

             Table "public.client_additional"
       Column   |  Type   | Collation | Nullable | Default
    ------------+---------+-----------+----------+---------
     client_id  | integer |           | not null |
     client_age | integer |           |          |
    Indexes:
        "pkey_ca" PRIMARY KEY, btree (client_id)
        "cov_idx" btree (client_id, client_age)

The client_id column in the client_additional table contains a subset of client's client_id values. Note the primary keys, and the other index I created on client_additional. I thought these would increase the benchmark query speed (see below) but it did not.

Importantly the db database is exactly the same (same structure, same values) except that it has no index or key .

Side note: the client and client_additional table should perhaps be a single table, since they give information at exactly the same level (client level). However, the database I'm using in real life came structured this way: some tables are split into several tables by "topic" although they give information at the same level. I don't know if that matters for my issue.

Benchmark Query

I'm using the following query, which mimics a lot what I need to do with real data:

    SELECT 
      client_additional.client_id, 
      client_additional.client_age,
      client.client_name
    FROM client
    INNER JOIN client_additional 
    ON client.client_id = client_additional.client_id;

Benchmark Results

On both databases, the benchmark query takes about 630 ms. Removing the keys and/or indexes in db_idx does not change anything. These benchmark results carry over to larger data sizes: speed is identical in the indexed and non-indexed cases.

That's where I am. How do I explain these results? Can I improve the join speed and how?

Answer 1

Use the EXPLAIN verb to see how the SQL engine intends to resolve the query. (Different SQL engines present this in different ways.) You can conclusively see whether the index will be used.

Also, you'll first need to load the tables with a lot of test data, because EXPLAIN will tell you what the SQL engine intends to do right now, and this decision is based in part on the size of the table and various other statistics. If the table is virtually empty, the SQL engine might decide that the index wouldn't be helpful now.

SQL engines use all kinds of very clever tricks to optimize performance, so it's actually rather difficult to get a useful timing test. But, if EXPLAIN tells you that the index is being used, that's pretty much the answer that you're looking for.

Answer 2

You have a primary key on the two tables which will be used for the join s. If you want to really see the queries slow down, remove the primary keys.

What is happening? Well, my guess is that the execution plans are the same with or without the secondary indexes. You would need to look at the plans themselves.

Unlike most other databases, Postgres does not get a benefit from covering indexes, because lock information is stored in the data pages only. So, the data pages always need to be accessed.

Answer 3

Setting up a small test DB, adding some rows and running your query:

CREATE TABLE client
(
   client_id integer PRIMARY KEY,
   client_name text
);

CREATE TABLE client_additional
(
   client_id integer PRIMARY KEY,
   client_age integer
);

INSERT INTO client (client_id, client_name) VALUES (generate_series(1,100000),'Phil');
INSERT INTO client_additional (client_id, client_age) VALUES (generate_series(1,70000),21);

ANALYZE;

EXPLAIN ANALYZE SELECT 
   client_additional.client_id, 
   client_additional.client_age,
   client.client_name
FROM
   client
INNER JOIN
   client_additional 
ON
   client.client_id = client_additional.client_id;

gave me this plan:

 Hash Join  (cost=1885.00..3590.51 rows=70000 width=11) (actual time=158.958..44 1.222 rows=70000 loops=1)
   Hash Cond: (client.client_id = client_additional.client_id)
   ->  Seq Scan on client  (cost=0.00..1443.00 rows=100000 width=7) (actual time =0.019..100.318 rows=100000 loops=1)
   ->  Hash  (cost=1010.00..1010.00 rows=70000 width=8) (actual time=158.785..15 8.786 rows=70000 loops=1)
         Buckets: 131072  Batches: 1  Memory Usage: 3759kB
         ->  Seq Scan on client_additional  (cost=0.00..1010.00 rows=70000 width =8) (actual time=0.016..76.507 rows=70000 loops=1)
 Planning Time: 0.357 ms
 Execution Time: 506.739 ms

What you can see from this is both tables were sequentially scanned, the values from each table were hashed and a hash join was done. Postgres determined this was the optimal way to execute this query.

If you were to recreate the tables without the Primary Key (and therefore remove the implicit index on the PK column of each), you get exactly the same plan, as Postgres has determined that the quickest way to execute this query is by ignoring the indexes and by hashing the table's values then doing a hash join on the two sets of hashed values to get the result.

After changing the number of rows in the client table like so:

TRUNCATE Client;

INSERT INTO client (client_id, client_name) VALUES (generate_series(1,200000),'phil');

ANALYZE;

Then I re-ran the same query and I see this plan instead:

Merge Join  (cost=1.04..5388.45 rows=70000 width=13) (actual time=0.050..415.50
3 rows=70000 loops=1)
   Merge Cond: (client.client_id = client_additional.client_id)
   ->  Index Scan using client_pkey on client  (cost=0.42..6289.42 rows=200000 width=9) (actual time=0.022..86.897 rows=70001 loops=1)
   ->  Index Scan using client_additional_pkey on client_additional  (cost=0.29..2139.29 rows=70000 width=8) (actual time=0.016..86.818 rows=70000 loops=1)
 Planning Time: 0.517 ms
 Execution Time: 484.264 ms

Here you can see that index scans were done, as Postgres has determined that this plan is a better one based on the current number of rows in the tables.

The point is that Postgres will use the indexes when it feels they will produce a faster result, but the thresholds before they are used are somewhat higher than you may have expected.

All best,

Phil