SQLite merging multiple rows with different column values into one entry

Question

I am trying to merge the column values of multiple rows into one entry using SQLite:

ID   date        test Val1  Val2 Val3 val4  Val5 Val6

S1   2020/01/01  ECT   5     5     5     0    0    0

S2   2020/02/01  FCT   2     2     2     0    0    0

S1   2020/02/02  ECT   0     0     0     5    5    5

S2   2020/03/01  FCT   0     0     0     2    2    2

the result should be:

 ID    test Val1  Val2 Val3 val4  Val5 Val6
    
 S1    ECT   5     5     5     5    5    5
    
 S2    FCT   2     2     2     2    2    2

I tried group_concat but I can't get it to work. I don't want to create a view, rather I want to reduce to number of incomplete or "half-completed" entries for the same ID in the DB.

Also, if it helps, I am using python to store the values in the SQLite DB. So if there is a way of doing it through python instead, that would also be great. Any help/comments are welcome

I am quite new to SQL so any advice would greatly be appreciated! Thank you

Answer 1

For this dataset, aggregation does what you want:

select id, test, max(val1) val1, max(val2) val2, max(val3) val3, max(val4) val4, max(val5) val5, max(val6) val6
from mytable
group by id, test

Answer 2

I am guessing that you have an aggregation query with date in the group by . Something like this:

select ID, date, test, . . .
from . . .
group by ID, date, test;

Change the query to remove date from both the select and group by :

select ID, test, . . .
from . . .
group by ID, test;

Answer 3

You can use Data Dictionary in order to select dynamically the columns with val prefix within the first auxiliary query :

import sqlite3
con = sqlite3.connect('db123.db')
cur = con.cursor()

query1 = "SELECT cl.name FROM sqlite_master sm CROSS JOIN pragma_table_info(sm.name) AS cl "
query1+= " WHERE sm.type='table' AND sm.name='tab' AND UPPER(cl.name) like 'VAL%' "
cur.execute(query1)
ls=cur.fetchall()

agr=''
col=''
for i in ls:
    col=str(i[0])
    agr+=",MAX("+col+") AS "+col

query2  = "SELECT ID, test {} FROM tab GROUP BY ID, test".format(agr)
cur.execute(query2)    

print(cur.fetchall())

con.close()

>>>

[('S1', 'ECT', 5, 5, 5, 5, 5, 5), ('S2', 'FCT', 2, 2, 2, 2, 2, 2)]