使用不同的参数多次运行同一查询

Question

I need to run a select SQL query from a .net service against a MYSQL DB. The query takes around 1 second to complete and needs to be executed 36 times consecutively with a different date for each run.

Simple example for the query, where date will change for each execution:

SELECT * FROM person where date < "some date"

I would like to know what are my options for running the query, and what is my best option performance wise. Should i run it 36 times against the DB? Use a stored procedure and loop through the different dates? Any other option?

Please note that there is no option to change the query to allow fewer executions, I must run it 36 times and i am trying to find out what are the viable options, pros and cons, for each option.

Edit:

I will try to make my query example clearer:

The query is comprised from several select statements, each select statements is making a calculation: either summing an amount or counting occurrences etc.. Each query is dependent on a date passed to the query. I need the results of those calculations for 36 different time periods.

The below example is not the original query but only a part of it with some changes for the names of the tables etc.. just to demonstrate the general idea.

I am currently running the query 36 times from my .Net server against my MYSQL DB. It just feels like this is not the best way to do this. I can consider moving the query to a stored procedures and perhaps running the same query in a loop 36 times instead of calling the DB for each query. I wanted to know if anyone has a better idea to tackle the issue of running the same query, with different parameters, many times.

Example:

SET @id = 11111;
SET @calculations_date = "2019-05-05";
SET @calculations_date_minus_1_year = DATE_SUB(@calculations_date, INTERVAL 1 YEAR);

SELECT customers.id,
IFNULL( (SELECT COUNT(DISTINCT id) FROM customer_data WHERE id = @id AND customer_data.date >= DATE_SUB(@calculations_date, INTERVAL 2 YEAR) AND customer_data.date <= @calculations_date) , 0) as customers_in_last_24_months,
IFNULL( (SELECT SUM(amount) FROM other_customer_data WHERE id = @id AND date <= @calculations_date_minus_1_year), 0) AS total_other_customer_data_until_12_months_before_date,
IFNULL( (SELECT SUM(amount) FROM other_customer_data2 WHERE id = @id AND date <= @calculations_date_minus_1_year), 0) AS total_other_customer_data2_until_12_months_before_date,
IFNULL( (SELECT SUM(amount) FROM other_customer_data3 WHERE id = @id AND date <= @calculations_date_minus_1_year), 0) AS total_other_customer_data3_until_12_months_before_date,
FROM customers
WHERE customers.id = @id;

Thanks!

Answer 1

Well, the first knee-jerk reaction to improving select-statement performance is to introduce an index to the table (in your case to the date column). Pro: quick, easy Con: more disk space required (depending on the kind of index and table size this can be considerable)

Another option that comes to my mind is to load the entire table into memory and do the filtering there. That is certainly faster, but especially for larger tables often not viable, since you might run out of RAM.

If there is any way to rewrite the query, you might want to select for all 36 people at once, but you stated, that "there is no option to change the query"... so I guess that option is out the window?

You may also experiment around with a materialized view, but I know too little about MySql to make any judgement on whether that makes much sense in your case.

Hope I was able to provide you with some options that you can uses as a starting point ;)

Answer 2

If there are a million rows before "some date", then it will take a long time to run, and your client will choke on the amount of data. So, I claim that this is not a realistic query.

On the other hand,

SELECT * FROM person where date < "some date"
    ORDER BY date  LIMIT 10

returns only (at most) 10 rows. If there is INDEX(date) , then the performance of the query will be very fast and nearly constant. I would expect milliseconds, not 1 second.

At that speed, you can have 360 users making the query "at the same time".

Or did you mean that one connection is doing 36 SELECTs . In this case it seems that they will be getting overlapping information??

The overhead is a significant of simple queries. Perhaps we can put the 36 queries into one?

Also, do you need all the columns from the table? Leave out unnecessary columns to cut back on the bulkiness of the transmission.

Answer 3

我决定在服务器端动态创建查询，并在每个部分之间包括UNION ALL以避免多次访问数据库。