Pivot Table in Mysql with a lot of dynamic columns

Question

I have problems in makeing a PIVOT Table in MySQL. I have following table. (it is a reduced demo table. The real one has 4000 stocks for fe 10 tradedates and 20 measurement values.)

     CREATE TABLE `levermann` (
       `RecNum` bigint(20) NOT NULL AUTO_INCREMENT,
       `Tradedate` date DEFAULT NULL,
       `Stock_Short` varchar(50) DEFAULT NULL,
       `Country` varchar(2) DEFAULT NULL,
       `LScore2` int(11) DEFAULT NULL,
       `MarketCAPUSD` bigint(20) DEFAULT NULL,
       PRIMARY KEY (`RecNum`)
     ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;

     -- ----------------------------
     -- Records of levermann
     -- ----------------------------
     INSERT INTO `levermann` VALUES ('8099', '2018-05-23', 'ANDR.VI', 'VI', '-9', '5109518494');
     INSERT INTO `levermann` VALUES ('8100', '2018-05-23', 'BWO.VI', 'VI', '-7', '4241189324');
     INSERT INTO `levermann` VALUES ('8101', '2018-05-23', 'CAI.VI', 'VI', '-7', '3222135865');
     INSERT INTO `levermann` VALUES ('8102', '2018-05-09', 'CWI.VI', 'VI', '-8', null);
     INSERT INTO `levermann` VALUES ('8103', '2018-05-23', 'EBS.VI', 'VI', '-7', '18317742129');
     INSERT INTO `levermann` VALUES ('8104', '2018-05-23', 'FLU.VI', 'VI', '-8', '3176359049');
     INSERT INTO `levermann` VALUES ('8105', '2018-05-23', 'IIA.VI', 'VI', '-8', '2767477473');
     INSERT INTO `levermann` VALUES ('8106', '2018-05-23', 'LNZ.VI', 'VI', '-9', '3027507195');

The output should be a table where each STOCKCODE (fe ANDR.VI) of these 8 should be a column with one selectable measurement value (fe LScore2) grouped by tradedate (= row).

I have found this Exmapl in MYSQL but I do not understand it completely. More over I have done this:

    SELECT
    tradedate, 
    GROUP_CONCAT(stock_short) as STOCKCODE
    FROM
    levermann
    GROUP BY
    Tradedate;

But here the stockcodes are in one cell and not in the header. Here is an image of an example of the wanted output. The total number of columns are about 4000 (max columns in table do not exceed 4096). And the tradedates (= rows) are about 350 days / years for 2 years in total.

It sheems clear that the columns should be created dynamically and can not hardcoded by AS statements.

Is there any solution for this difficult question? THANKS a lot.

Update: I think on a dynamically running statement like this..

    SET @sql = NULL;

    SELECT GROUP_CONCAT(concat(LScore2,' AS `LScore_',Stock_Short,'`')) into 
    @sql from levermann;

    SET @sql = CONCAT('SELECT tradedate, ', @sql, '
              FROM levermann 
               GROUP BY tradedate');

     PREPARE stmt FROM @sql;
     EXECUTE stmt;
     DEALLOCATE PREPARE stmt;

But I do not know how I can create the string:

LScore AS LScore_ANDR.VI, LScore AS LScore_BWO.VI, ...

I really in doubt but it might be done in a way in MySQL. Of course I can make a php script too. But I would like to learn how it is done in MySQL.

Update 2: I think I could make it. I am not quite sure if it is correct, but it is dynamically created. In MySQL

Answer 1

I think I have got it:

SET SESSION group_concat_max_len = @@max_allowed_packet;
SET @sql = NULL;

SELECT GROUP_CONCAT(concat('MAX(CASE Stock_Short WHEN \'',Stock_Short,'\'   THEN \'',LScore2,'\' END) AS `LScore_',Stock_Short,'`')) into @sql from levermanndemo where country = 'VI';

SET @sql = CONCAT('SELECT tradedate, ', @sql, '
              FROM levermanndemo  
               GROUP BY tradedate');


PREPARE stmt FROM @sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;

OUTPUT:

Update: Unfortunatly it makes troubel if the table looks like this.

I tried this solution:

    SET SESSION group_concat_max_len = @@max_allowed_packet;
    SET @sql = NULL;

    SELECT  GROUP_CONCAT( DISTINCT concat('MAX(CASE WHEN p.Stock_Short = 
    \'',f.Stock_Short,'\' AND `Tradedate` = \'', f.tradedate,'\'  THEN 
    \'',f.LScore2,'\' ELSE NULL END) AS   `LScore_',f.Stock_Short,'`')) 
    into @sql from  levermanndemo f ;

    SET @sql = CONCAT('SELECT p.tradedate, ', @sql, ' FROM levermanndemo p 
    GROUP BY p.tradedate');
    #SELECT @sql; 

    PREPARE stmt FROM @sql;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;

The output is not desired - and wrong. There are now 2 columns with the same STOCK (BWO.VI and BWO.VI1) , I would like to merge these two columns together. But how can it be done?

Answer 2

In SQL, you can't write any query that expands columns dynamically based on the data values it finds once it starts reading data. All the columns must be fixed in the query's select-list at the time you prepare the query -- before it reads any data.

This means you must know all the distinct values, and you need to produce a select-list with one column for each value you want to include in the pivot.

You can solve this by doing two queries: one to fetch all the distinct stock values:

SELECT DISTINCT Stock_Short FROM levermann

Then based on that result, format a looooong SQL query with on expression for each stock value. Start with the column you know you want that is not one of the dynamic stock-related columns:

SELECT Tradedate,

Then for each row in the result from the firs query, append a column like:

MAX(CASE Stock_Short WHEN <value> THEN LScore2 END) AS <alias>,

Then finally append the end of the query:

FROM levermann
GROUP BY Tradedate;

My recommendation since you have 4000 distinct stock values is that you should just fetch the data from the database as-is, and use application code to present a pivoted display. That is, loop over the result of the SQL query, all 4000 rows (not columns), and arrange it into an object in your application space. Then format that object however you want for display.

Answer 3

If the list of StockCodes is fixed can one not simply fix the columns of the new table and then filter for the data in the original table to copy into the new table?

At least this would be the process when doing it manually in Excel for instance.