如何从 Postgresql 表中的现有数据分片

Question

I have a large table inter , which contains 50 billion rows. Each row consists of two columns, both of them are actually foreign keys of IDs of the other two tables(just the relation, foreign key constraints were not set in the database).

My table structure is like:

create table test_1(
    id integer primary key,
    content varchar(300),
    content_len integer
);

create index test_1_id_len on test_1(id, content_len);

--this has 1.5 billion rows.
-- example row1: 1, 'alskfnla', 8
-- example row2: 1, 'asdgaagder', 10
-- example row3: 1, 'dsafnlakdsvn', 12

create table test_2(
    id integer primary key,
    split_str char(3)
);

--this has 60,000 rows.
-- example row1: 1, 'abc'
-- example row2: 2, 'abb'

create table inter(
    id_1 integer,    -- id of test_1
    id_2 integer     -- id of test_2 
);

create index test_index_1 on inter(id_1);
create index test_index_2 on inter(id_2);
create index test_index_1_2 on inter(id_1, id_2);

--this has 50 billion rows.
-- example row1: 1, 2
-- example row2: 1, 3
-- example row3: 1, 4

Further, I need to do some queries like

select * 
from inter 
  inner join test_1 on(test_1.id = inter.id_1) 
where id_2 in (1,2,3,4,5,67,8,9,10) 
  and test_1.content_len = 30 
order by id_2;

The reason why I want to shard the table is that I could not create indices on the two columns( the transactions did not end for one week, and it occupied full virtual memory).

SO I am considering to shard the table by one of the columns. This column has around 60,000 values, from 1 to 60,000. I would like to shard the table to 60,000 subtables. I do some searches, but most of the articles do it by a trigger, which could not be applied in my case since the data are already in the table. Does anyone know how to do that, thanks a lot!

ENV: redhat, RAM 180GB, postgresql 11.0

Answer 1

You don't want to shard the table, but partition it.

60000 partitions is too many. Use list partitioning to split the table in something like at most 600 partitions. Make sure to upgrade to PostgreSQL v12 so that you can benefit from the latest performance improvements.

The hard part will be moving the data without eexcessive downtime. Perhaps you can use triggers to capture changes while you INSERT INTO ... SELECT and catch up later.