使用大型数据集加快Postgresql查询

Question

I have a postgresql database which has 2 tables that I am interested in at the moment. The first table is my "file" table, which contains a file name, some relevant information about that file, and has a serial id as its primary key. Here is a rough outline of my file table:

fileData(fileName varchar(120) unique, ... other info, id serial primary key)

I then have another table that contains information from the files listed in the file table. It is linked to the file table though the id of the file table. There is a variable number of lines in the "data" table corresponding to each file, with the line numbers varying from several hundred to several hundred thousand. Here is a rough outline of my data table:

rawData(fileID integer references fileData(id), lineNum integer, data1 double, ... other info)

To go with the above, I have a query where I first sort through the fileData to get the id of each file, as well as some of the other info. Then I am looking to sort through the raw data corresponding to that file to find "interesting" information. This particular query is written in c++ using Qt to handle the actual processing, but the majority of the work is being done by the database (and Qt just passes the database query in as a text query and that query needs to match all of the formatting that the sql database would normally need). Below is an example of my query:

QSqlQuery fileQuery, dataQuery;
int id;
fileQuery.prepare("SELECT id, fileType FROM fileData ORDER BY id");
if (!fileQuery.exec()){
    //error
    return;
}
while (fileQuery.next()){
    id = fileQuery.value(0).toInt();
    dataQuery.prepare("Select lineNum, data1, ...other info "
                      "FROM rawData WHERE fileID = ? and data1 < ? "
                      "ORDER BY fileID, lineNum");
    dataQuery.addBindValue(id);
    dataQuery.addBindValue(num);
    if (!dataQuery.exec()){
        return;
    }
    while (dataQuery.next()){
      //code to load pertinant info into my program to handle later
    }
}

This program took about 2 hours or so to run up until recently, with 1400 files loaded and about a million or so lines of data. However, I just got a bunch more data, and now am up to 1650 files of data, with 130 million lines of data, and my program has slowed to a crawl. What used to take two hours has now taken over 6 to go through only 1/4 of the files that I now have, and my debug output has told me that I am still working through files that I have run this program on previously, not any new data yet. Checking my task manager, I can see that my program is barely working, while postgresql is using an entire core to give me the data I am asking, so I know that the current hold up is in my sql commands, not in what I am doing with the data in the meantime.

Lastly, at the moment, throwing more hardware at the problem is not something that I can do. With that being said, is there anything that I can do to optimize my queries to increase the speed at which I am accessing this data? Or am I already doing things correctly, and just have to suck it up and deal with the slowness due to the size of the data set that I am working with?

Answer 1

You probably could only execute each query only once.

1) The file table is so small that you can load it in a memory map and be done with it

2) The query on the data table, filtered by fileType and ordered by file id, should not take ages (of course you have an index on fileID + lineNum, right ?)

Answer 2

Is there any reason not to combine the two queries into one?

SELECT id, fileType, lineNum, data1, ...other info 
FROM fileData LEFT JOIN rawData on fileData.id = rawData.fileID
WHERE data1 < ? 
ORDER BY fileID, lineNum

Also since you say num is a constant in the function, instead of binding it to a replaceable parameter, I'd just construct the query string with its value. Making sure you have the right indexes on both tables is imperative also.