如何有效地将基于时间的数据集分组以计算组平均值和中位数

Question

I have a table with 100000+ records as below. According to the system, this table might have daily transactions and it will be growing day by day.

+---------------------+-----------+
|        Date         | Value     |
+---------------------+-----------+
| 2018-12-21 11:17:00 | 85.8      |
| 2018-12-28 15:07:00 | 16.2      |
| 2019-01-28 08:05:00 | 24.8      |
| 2019-02-28 12:07:00 | 13.9      |
| 2019-05-28 10:48:00 | 8         |
| 2019-05-28 09:17:00 | 40.6      |
| 2019-08-28 10:06:00 | 71.9      |
| 2019-08-16 17:28:00 | 36        |
| 2019-08-28 10:07:00 | 1922      |
| …                   | …         |
+---------------------+-----------+

I want to group the data by quarters and get the quarterly average and median to show in graphs as follows.

Average - example graph average X= Quarter, Y = Value

Median - example graph median X= Quarter, Y = Value

I am using PHP Laravel 5.8 and a MySQL database.

In my approach, first I created an array of keys [year-quarter] depending on the start date and end date [“2018 3”, “2018 4”, “2019 1”, “2019 2”, “2019 3”]

And then I used a foreach loop to read through the 100000+ records and put values in subarrays under relevant key. This hits 100% CPU usage in an apache, 2 core server with 4GB RAM for one user access. The foreach loop was observed to be consuming a huge CPU power.

// prepare all array keys        
$chunkedData = array();
while (Carbon::parse($startDate)<=Carbon::parse($endDate)) {
        $chunkedData[Carbon::parse($startDate)->isoFormat('Y Q')] = array();
            $startDate = Carbon::parse($startDate)->addMonths(3);
        }
// foreach loop to read all the records
foreach ($arrScanData as $scanData) {
    $key = Carbon::parse($scanData->qr_generated_date)->isoFormat('Y Q');
    array_push($chunkedData[$key], (float)$scanData->value);
}

I appreciate if you can give me few solutions (logical and architecture) to overcome my problem.

Answer 1

I would let MySQL do the bulk of the processing.

The following would give you the quarters with their values:

SELECT CONCAT(YEAR(date) , ' ', QUARTER(date)) AS quarter, value FROM yourtable

The averages could be calculated directly with:

SELECT CONCAT(YEAR(date) , ' ', QUARTER(date)) AS quarter, AVG(value) AS average FROM yourtable GROUP BY quarter

The median calculation would take some processing based on the first query. I have no experience in that. MariaDB does have a MEDIAN function.

Answer 2

I would suggest you to change your approach as bellow.

Do not read 100000+ records once. Just read start and end date from the table to build your quarter array. Then you can loop $chunkedData array and read data for each quarter by providing quarter start and end dates or may be just with quarter.