Split a (slow) mysql query in chunks of many primary_key ranges (id 1-100, id 101-200, ...)

Question

Using mariadb version 10.5.15, Flask 2.1.2, Flask-Session 0.2.0 and python 3.9

My problem is a very slow query of my database table. Especially when the result of the query contains many rows, it takes too long.

Now I have the idea to split every query up in many queries, by just including an extra "WHERE" part like this:

• Query 1: Select ... FROM ... WHERE ... AND primary_key > 0 AND primary_key <= 100 ...
• Query 2: Select ... FROM ... WHERE ... AND primary_key > 100 AND primary_key <= 200 ...
• Query 3: Select ... FROM ... WHERE ... AND primary_key > 200 AND primary_key <= 300 ...
• ...

My question is if there is a better way or if this attempt brings other problems (or not)

The result of the query will afterwards be an answer to a POST request. And because I'm using sessions, I would continuously push results of each chunked query in a session variable "search_query: list".

While this search_query gets filled I check if there are enough results (or if the query is done the query is done) I sent a first chunk as response.

My guess:

1. I have to run the db query in a separate process (an extra thread with python)
2. I need a stream connection with whoever made the request
3. I won't have problems with accessing the same variable (with two processes), because only one process writes to it and I never remove elements from the variable

This is the query:

WITH pf 
AS (
SELECT short_description, long_description, count, unit, cost_characteristic, size, family_id, is_parent  
FROM positions 
INNER JOIN project_crafts 
USING (project_craft_id) 
INNER JOIN projects 
USING (project_id) 
WHERE ---dynamic WHERE conditions---
) 
SELECT positions.short_description, positions.long_description, positions.count, positions.unit, positions.cost_characteristic, project_crafts.size, positions.family_id, positions.is_parent  
FROM positions 
INNER JOIN pf 
ON positions.family_id=pf.family_id 
AND COALESCE(positions.is_parent, -1) <> COALESCE(pf.is_parent, -1) 
INNER JOIN project_crafts 
USING (project_craft_id) 
UNION SELECT short_description, long_description, count, unit, cost_characteristic, size, family_id, is_parent  
FROM pf

The query is slow, when ---dynamic WHERE conditions--- is empty. Then the query returns every single row and the JOINs get expensive (I think).

Answer 1

讓我們看看查詢的 rest... 你在取什么列； WHERE 里還有更多的東西嗎？

 SELECT CEILING(pk/100) * 100 AS 'through', COUNT(*) AS 'how many', SUM(foo) FROM... WHERE... GROUP BY CEILING(pk/100) ORDER BY CEILING(pk/100)