Query from multiple table in MySQL and sum up

Question

I have seven tables (from XT1 to XT7) and each have the same structure as below:

---

Date	m1	m2	m3
2021-06-01	4	2	6
2021-06-02	3	2	5
2021-06-03	12	2	14
.....	..	..	..

I only need m3 value from each table. What I'm trying to do, is to show all m3 values in a single table and sum them up like below:

Date	XT1.m3	XT2.m3	XT3.m3	XT4.m3	XT5.m3	XT6.m3	XT7.m3	Subtotal XT1~7.m3
2021-06-01	6	7	8	6	7	8	8	50
2021-06-02	6	7	8	6	7	8	8	50
2021-06-03	6	7	8	6	7	8	8	50
Total	18	21	24	18	21	24	24	150

What I have tried before:

select (TX1.m3+TX2.m3+TX3.m3...) AS subtotal, date_format(date, '%Y-%m') as date,TX1.m3,TX2.m3,TX3.m3
from TX1 AS c1 left join TX2 AS c2 on TX1.date=TX2.date
left join TX3 AS c3 on TX2.date=TX3.date

Which was modified from a previous working script. But the old structure only had 3 tables and now I have seven.

UPDATE:Tried the following

SELECT TX1.date,TX1.m3,TX2.m3,TX2.m3...TX7.m3
FROM TX1
inner JOIN TX2 ON TX1.date = TX2.date
inner JOIN TX3 ON TX2.date = TX3.date
......
inner JOIN TX7 ON TX6.date = TX7.date

Above code will return an un-grouped matrix. Tried to group with 'date' and returned error 1055. Override with

SET sql_mode=(SELECT REPLACE(@@sql_mode,'ONLY_FULL_GROUP_BY',''));

Now works, but not sure if this is the correct way to do. Any ideas?

Answer 1

If you want to use the same query structure with addition of newly created tables, you can try using prepared statement. For example:

Preparing field values:

/*setting variables as NULL*/
SET @sql = NULL;
SET @tbl = NULL;
SET @val1 = NULL;
SET @val2 = NULL;
SET @reftbl = 'TX1'; /*this is your reference (most left) table in the LEFT JOIN*/

/*setting each variables with values*/

SELECT GROUP_CONCAT(CONCAT(table_name,'.',column_name,' AS ',table_name,column_name) SEPARATOR ', ') INTO @val1
FROM information_schema.columns 
WHERE table_name LIKE 'TX%'
#AND table_schema=your_database_name
AND column_name='m3';

SELECT CONCAT(GROUP_CONCAT(CONCAT(table_name,'.',column_name) ORDER BY table_name SEPARATOR '+' ),' as subtotal') INTO @val2
FROM information_schema.columns 
WHERE table_name LIKE 'TX%'
#AND table_schema=your_database_name
AND column_name='m3'

Here we are using information_schema.columns tables to generate field of all the m3 columns coming from all tables that correspond to the condition ( table_name and table_schema ). This will return you something like:

@val1: TX1.m3 AS TX1m3, TX2.m3 AS TX2m3, TX3.m3 AS TX3m3, TX4.m3 AS TX4m3, TX5.m3 AS TX5m3, TX6.m3 AS TX6m3, TX7.m3 AS TX7m3

@val2:TX1.m3+TX2.m3+TX3.m3+TX4.m3+TX5.m3+TX6.m3+TX7.m3 as subtotal

Next is to prepare the tables required in the query and add LEFT JOIN .

SELECT CONCAT(@reftbl, 
          GROUP_CONCAT(
               CASE WHEN tbn IS NOT NULL 
               THEN CONCAT(' LEFT JOIN ', table_name,' ON ',tbn,'.date =',table_name,'.date') 
          ELSE table_name END SEPARATOR ' ')) INTO @tbl
FROM
(SELECT @reftbl tbn,
       table_name 
FROM information_schema.tables 
WHERE table_name LIKE 'TX%'
#AND table_schema=your_database_name
AND table_name <> @reftbl) v;

The operation I'm doing here is to generate the LEFT JOIN parts by using CONCAT and GROUP_CONCAT .

Then we set @sql variable with a query generated based on all the variables we previously set; constructed using CONCAT .

/*constructing query and set into @sql*/
SET @sql = CONCAT('SELECT DATE_FORMAT(TX1.DATE, "%Y-%m") AS DATE,',@val1,',',@val2,'
               FROM ',@tbl,';'); 

This will end up with a query like:

SELECT DATE_FORMAT(TX1.DATE, "%Y-%m") AS DATE,TX1.m3 AS TX1m3, TX2.m3 AS TX2m3, 
       TX3.m3 AS TX3m3, TX4.m3 AS TX4m3, TX5.m3 AS TX5m3, TX6.m3 AS TX6m3, TX7.m3 AS TX7m3,
       TX1.m3+TX2.m3+TX3.m3+TX4.m3+TX5.m3+TX6.m3+TX7.m3 as subtotal
FROM TX1 LEFT JOIN TX2 ON TX1.date =TX2.date  
LEFT JOIN TX3 ON TX1.date =TX3.date  
LEFT JOIN TX4 ON TX1.date =TX4.date  
LEFT JOIN TX5 ON TX1.date =TX5.date  
LEFT JOIN TX6 ON TX1.date =TX6.date  
LEFT JOIN TX7 ON TX1.date =TX7.date;

Lastly, prepare, execute then deallocate the @sql statement and you'll get the desired result you're looking for:

PREPARE stmt FROM @sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt; 

This is a way of generating 'dynamic query' which means if/when you add/remove tables, the query will be generated with the additional or without the removed tables. For example, if there's a new table TX8 , when you run the queries above, it will include TX8 in the prepared statement as long as it match the condition.

Here's a demo fiddle that includes a situation when there's a new table created.

Answer 2

In another database, this problem would be solved with multiple FULL joins, but since MySql does not support them and simulating them for 7 tables is not an option, I would choose UNION ALL .

From each table, return 8 columns: 1 for the date, 1 for m3 and 6 null columns.
Use UNION ALL to combine them and finally aggregate:

SELECT date, 
       MAX(m3_1) m3_1, MAX(m3_2) m3_2, MAX(m3_3) m3_3, MAX(m3_4) m3_4, MAX(m3_5) m3_5, MAX(m3_6) m3_6, MAX(m3_7) m3_7,
       COALECSE(MAX(m3_1), 0) + COALECSE(MAX(m3_2), 0) + COALECSE(MAX(m3_3), 0) + COALECSE(MAX(m3_4), 0) + 
       COALECSE(MAX(m3_5), 0) + COALECSE(MAX(m3_6), 0) + COALECSE(MAX(m3_7), 0) subtotal
FROM (
  SELECT date, m3 m3_1, null m3_2, null m3_3, null m3_4, null m3_5, null m3_6, null m3_7 FROM XT1
  UNION ALL
  SELECT date, null, m3, null, null, null, null, null FROM XT2
  UNION ALL
  SELECT date, null, null, m3, null, null, null, null FROM XT3
  UNION ALL
  ............................................................
  SELECT date, null, null, null, null, null, null, m3 FROM XT7
) t
GROUP BY date