批处理/拆分PostgreSQL数据库

Question

I am working on a project which processes data in batches and fills up a PostgreSQL (9.6, but I could upgrade) database. The way it currently works is that the process happens in separate steps and each step adds data to a table that it owns (rarely two processes write in the same table, if they do, they write in different column).

The way the data happens to be, the data tends to become more and more fine-grained with each step. As a simplified example I have one table defining the data sources. There are very few (in the tens/ low hundreds), but each of these data sources generate batches of data samples (batches and samples are separate tables, to store metadata). Each batch typically generates about 50k samples. Each of these data points then gets processed step-by-step and each data sample generates more data-points in the next table.

This worked fine, until we got to a 1.5mil rows in the sample table (which is not a lot of data from our point of view). Now filtering for a batch starts becoming slow (about 10ms for each sample we retrieve). And it starts becoming a major bottleneck, because the execution time to get the data for a batch take 5-10mins (fetching is ms).

We have b-tree indices on all foreign keys that are involved for these queries.

Since our computations target the batches, I do normally not need to query across batches during the computation (this is when the query time hurts a lot at the moment). However for data-analysis reasons ad-hoc queries across batches need to remain possible.

So a very simple solution would be to generate an individual database for each batch, and somehow query across these databases when I need to. If I had only one batch in each database, obviously the filtering for a single batch would be instant and my problem would be solved (for now). However, then I would end up with thousands of databases and the data-analysis would be painful.

Within PostgreSQL, is there a way of pretending that I have separate databases for some queries? Ideally I would like to do that for each batch when I "register" a new batch.

Outside of the world of PostgreSQL, is there another database I should try for my usecase?

Edit: DDL / Schema

In our current implementation, sample_representation is the table that all processing results depend on. A batch is truly defined by a tuple of ( batch.id , representation.id ). The query I tried and described above as slow is (10ms for each sample, adding up to around 5 min for 50k samples)

SELECT sample_representation.id, sample.sample_pos
FROM sample_representation
JOIN sample ON sample.id = sample_representation.id_sample
WHERE sample_representation.id_representation = 'representation-uuid' AND sample.id_batch = 'batch-uuid'

We have currently somewhere around 1.5 s sample s, 2 representation s, 460 batch es (of which 49 have been processed, the others do not have samples associated to it), which means each batch has 30k samples in average. Some have around 50k.

The schema is below. There is some meta-data associated with all tables, but I am not querying for it in this case. The actual sample-data are stored separately on disk and not in the database, in case that makes a difference.

    create table batch
(
    id uuid default uuid_generate_v1mc() not null
        constraint batch_pk
            primary key,
    path text not null
        constraint unique_batch_path
            unique,
    id_data_source uuid
)
;
create table sample
(
    id uuid default uuid_generate_v1mc() not null
        constraint sample_pk
            primary key,
    sample_pos integer,
    id_batch uuid
        constraint batch_fk
            references batch
                on update cascade on delete set null
)
;
create index sample_sample_pos_index
    on sample (sample_pos)
;
create index sample_id_batch_sample_pos_index
    on sample (id_batch, sample_pos)

;
create table representation
(
    id uuid default uuid_generate_v1mc() not null
        constraint representation_pk
            primary key,
    id_data_source uuid
)
;
create table data_source
(
    id uuid default uuid_generate_v1mc() not null
        constraint data_source_pk
            primary key
)
;
alter table batch
    add constraint data_source_fk
        foreign key (id_data_source) references data_source
            on update cascade on delete set null
;
alter table representation
    add constraint data_source_fk
        foreign key (id_data_source) references data_source
            on update cascade on delete set null
;
create table sample_representation
(
    id uuid default uuid_generate_v1mc() not null
        constraint sample_representation_pk
            primary key,
    id_sample uuid
        constraint sample_fk
            references sample
                on update cascade on delete set null,
    id_representation uuid
        constraint representation_fk
            references representation
                on update cascade on delete set null
)
;
create unique index sample_representation_id_sample_id_representation_uindex
    on sample_representation (id_sample, id_representation)
;
create index sample_representation_id_sample_index
    on sample_representation (id_sample)
;
create index sample_representation_id_representation_index
    on sample_representation (id_representation)
;

Answer 1

After fiddling around, I found a solution. But I am still not sure why the original query really takes that much time:

SELECT sample_representation.id, sample.sample_pos
FROM sample_representation
JOIN sample ON sample.id = sample_representation.id_sample
WHERE sample_representation.id_representation = 'representation-uuid' AND sample.id_batch = 'batch-uuid'

Everything is indexed, but the tables are relatively big with 1.5 million rows in sample_representation and in sample . I guess what happens is that first the tables get joined and then filtered with WHERE . But even if creating a large view as a result of the join, it should not take that long?!

In any case, I tried to use a CTE instead of joining two "massive" tables. The idea was to filter early and then join afterwards:

WITH sel_samplerepresentation AS (
  SELECT *
  FROM sample_representation
  WHERE id_representation='1437a5da-e4b1-11e7-a254-7fff1955d16a'
  ), sel_samples AS (
  SELECT *
  FROM sample
  WHERE id_video='75c04b9c-e4b9-11e7-a93f-132baa27ac91'
)
SELECT sel_samples.sample_pos, sel_samplerepresentation.id
FROM sel_samplerepresentation
JOIN sel_samples ON sel_samples.id = sel_samplerepresentation.id_representation

This query also takes forever. Here the reason is clear. sel_samples and sel_samplerepresentation have 50k records each. The join happens on a non-indexed column of the CTEs.

Since there are no indices for CTEs, I reformulated them as materialized views for which I can add indices:

CREATE MATERIALIZED VIEW sel_samplerepresentation AS (
  SELECT *
  FROM sample_representation
  WHERE id_representation='1437a5da-e4b1-11e7-a254-7fff1955d16a'
  );

CREATE MATERIALIZED VIEW sel_samples AS (
  SELECT *
  FROM sample
  WHERE id_video = '75c04b9c-e4b9-11e7-a93f-132baa27ac91'
);

CREATE INDEX sel_samplerepresentation_sample_id_index ON sel_samplerepresentation (id_sample);
CREATE INDEX sel_samples_id_index ON sel_samples (id);

SELECT sel_samples.sample_pos, sel_samplerepresentation.id
FROM sel_samplerepresentation
JOIN sel_samples ON sel_samples.id = sel_samplerepresentation.id_sample;

DROP MATERIALIZED VIEW sel_samplerepresentation;
DROP MATERIALIZED VIEW sel_samples;

This is more of a hack than a solution, but executing these queries takes 1s! (down from 8min)