MariaDB 10.5.11 slow performance compared to MySQL 8.0.13

Question

Having trouble with particular query performance after migrating large database from MySQL 8.0.13 data to MariaDB 10.5.11.

I install MariaDB via repo:

[mariadb]
name = MariaDB 10.5 para CentOS 7
baseurl = http://yum.mariadb.org/10.5/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

To export database I use: mysqldump database > database.sql

I have also recreated the indexes after import.

New server:

• centos-release-7-9.2009.1.el7.centos.x86_64
• 16GB RAM
• 8 CPU

Old server:

• centos-release-6-10.el6.centos.12.3.x86_64
• 12 GB RAM
• 4 CPU

SQL query with slow performance:

SELECT  rep.unique_id AS report_id, rep.name AS report_title,
        sir.slide_id, s.slide_order, st.tag_category_id, stc.txt as categoria,
        st.tag_value_id, stv.txt as valor, count(sir2.slide_id) as Slides,
        count(s2.media_id) as Medias
    FROM  er_reports rep
    INNER JOIN  er_slides_in_report sir  ON sir.deleted_date IS NULL
              AND  rep.unique_id = sir.report_id
    INNER JOIN  er_slides_in_report sir2  ON sir.deleted_date IS NULL
              AND  rep.unique_id = sir2.report_id
    INNER JOIN  er_slides s  ON s.deleted_date IS NULL
              AND  s.unique_id = sir.slide_id
    LEFT JOIN  er_slides s2  ON s2.deleted_date IS NULL
              AND  sir2.slide_id = s2.unique_id
    LEFT JOIN  er_media m  ON m.deleted_date IS NULL
              AND  s2.media_id = m.unique_id
    INNER JOIN  er_slides_tags st  ON st.deleted_date IS NULL
              AND  s.unique_id = st.slide_id
    INNER JOIN  er_slide_tags_categories stc  ON st.tag_category_id = stc.id
    INNER JOIN  er_slide_tags_values stv  ON st.tag_value_id = stv.id
    WHERE  rep.deleted_date IS NULL
      AND  s.slide_order < 0
      AND  rep.user_id = 1606741093465
    GROUP BY  rep.unique_id, sir.slide_id, st.tag_category_id,
              st.tag_value_id;

Explain MariaDB:

+------+-------------+-------+---------------+-----------------------------------------------------+---------------------------+---------+----------------------------------+---------+---------------------------------+
| id   | select_type | table | type          | possible_keys                                       | key                       | key_len | ref                              | rows    | Extra                           |
+------+-------------+-------+---------------+-----------------------------------------------------+---------------------------+---------+----------------------------------+---------+---------------------------------+
|    1 | SIMPLE      | stc   | index         | PRIMARY                                             | txt                       | 767     | NULL                             | 47      | Using index                     |
|    1 | SIMPLE      | st    | ref           | slide_id,tag_value_id,slide_cat_val,tag_category_id | tag_category_id           | 8       | cloud_discipline.stc.id          | 376     | Using where                     |
|    1 | SIMPLE      | stv   | eq_ref        | PRIMARY                                             | PRIMARY                   | 8       | cloud_discipline.st.tag_value_id | 1       |                                 |
|    1 | SIMPLE      | sir   | ref           | report_id,slide_id,deleted_date,slide_in_report     | slide_id                  | 194     | cloud_discipline.st.slide_id     | 1       | Using where                     |
|    1 | SIMPLE      | rep   | eq_ref|filter | unique_index,user_id,deleted_date                   | unique_index|deleted_date | 194|6   | cloud_discipline.sir.report_id   | 1 (49%) | Using where; Using rowid filter |
|    1 | SIMPLE      | s     | ref           | unique_id,deleted_date                              | unique_id                 | 194     | cloud_discipline.st.slide_id     | 1       | Using where                     |
|    1 | SIMPLE      | sir2  | ref           | report_id,slide_in_report                           | slide_in_report           | 194     | cloud_discipline.sir.report_id   | 17      | Using index                     |
|    1 | SIMPLE      | s2    | ref           | unique_id,deleted_date                              | unique_id                 | 194     | cloud_discipline.sir2.slide_id   | 1       | Using where                     |
|    1 | SIMPLE      | m     | ref           | unique_id,deleted_date                              | unique_id                 | 194     | cloud_discipline.s2.media_id     | 1       | Using where                     |
+------+-------------+-------+---------------+-----------------------------------------------------+---------------------------+---------+----------------------------------+---------+---------------------------------+

Explain MySql:

+----+-------------+-------+------------+--------+--------------------------------------------------------+-----------------+---------+---------------------------------+--------+----------+-----------------------+
| id | select_type | table | partitions | type   | possible_keys                                          | key             | key_len | ref                             | rows   | filtered | Extra                 |
+----+-------------+-------+------------+--------+--------------------------------------------------------+-----------------+---------+---------------------------------+--------+----------+-----------------------+
|  1 | SIMPLE      | sir   | NULL       | ref    | report_id,slide_id,deleted_date,slide_in_report        | deleted_date    | 6       | const                           | 192448 |   100.00 | Using index condition |
|  1 | SIMPLE      | rep   | NULL       | eq_ref | unique_id,unique_id_2,unique_id_3,user_id,deleted_date | unique_id       | 194     | cloud_soccer.sir.report_id      |      1 |     5.00 | Using where           |
|  1 | SIMPLE      | s     | NULL       | ref    | unique_id,deleted_date                                 | unique_id       | 194     | cloud_soccer.sir.slide_id       |      1 |    16.66 | Using where           |
|  1 | SIMPLE      | st    | NULL       | ref    | slide_id,tag_value_id,slide_cat_val,tag_category_id    | slide_cat_val   | 194     | cloud_soccer.sir.slide_id       |     17 |    10.00 | Using where           |
|  1 | SIMPLE      | stc   | NULL       | eq_ref | PRIMARY                                                | PRIMARY         | 8       | cloud_soccer.st.tag_category_id |      1 |   100.00 | NULL                  |
|  1 | SIMPLE      | stv   | NULL       | eq_ref | PRIMARY                                                | PRIMARY         | 8       | cloud_soccer.st.tag_value_id    |      1 |   100.00 | NULL                  |
|  1 | SIMPLE      | sir2  | NULL       | ref    | report_id,slide_in_report                              | slide_in_report | 194     | cloud_soccer.sir.report_id      |     35 |   100.00 | Using index           |
|  1 | SIMPLE      | s2    | NULL       | ref    | unique_id,deleted_date                                 | unique_id       | 194     | cloud_soccer.sir2.slide_id      |      1 |   100.00 | Using where           |
|  1 | SIMPLE      | m     | NULL       | ref    | unique_id,unique_id_2,deleted_date                     | unique_id       | 194     | cloud_soccer.s2.media_id        |      1 |   100.00 | Using where           |
+----+-------------+-------+------------+--------+--------------------------------------------------------+-----------------+---------+---------------------------------+--------+----------+-----------------------+

MariaDB: 2-3 min

MySQL: 1.59sec

MariaDB's my.cfg

[mysqld]
bind-address = 0.0.0.0
port = 3306
collation_server=utf8_general_ci
character_set_server=utf8

skip_name_resolve
innodb_buffer_pool_size = 8600MB
innodb_log_file_size=2048MB
max_allowed_packet=64M
performance_schema = ON
table_definition_cache = -1

MySqlTuner:

 >>  MySQLTuner 1.8.1 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.pl/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[!!] Successfully authenticated with no password - SECURITY RISK!
[OK] Currently running supported MySQL version 10.5.11-MariaDB
[OK] Operating on 64-bit architecture

-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/log/mysqld.log exists
[--] Log file: /var/log/mysqld.log(0B)
[--] Log file /var/log/mysqld.log is empty. Assuming log-rotation. Use --server-log={file} for explicit file

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE
[--] Data in MyISAM tables: 13.0M (Tables: 11)
[--] Data in InnoDB tables: 13.0G (Tables: 208)
[OK] Total fragmented tables: 0

-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] User 'webappuser'@% does not specify hostname restrictions.
[--] There are 620 basic passwords in the list.

-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 39m 49s (2K q [1.134 qps], 1K conn, TX: 771K, RX: 720K)
[--] Reads / Writes: 100% / 0%
[--] Binary logging is disabled
[--] Physical Memory     : 15.5G
[--] Max MySQL memory    : 18.6G
[--] Other process memory: 0B
[--] Total buffers: 8.8G global + 66.9M per thread (151 max threads)
[--] P_S Max memory usage: 72B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 8.9G (57.46% of installed RAM)
[!!] Maximum possible memory usage: 18.6G (120.22% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (0/2K)
[OK] Highest usage of available connections: 1% (2/151)
[OK] Aborted connections: 0.00%  (0/1340)
[OK] Query cache is disabled by default due to mutex contention on multiprocessor machines.
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 4 sorts)
[OK] No joins without indexes
[OK] Temporary tables created on disk: 9% (6 on disk / 63 total)
[OK] Thread cache hit rate: 99% (4 created / 1K connections)
[OK] Table cache hit rate: 70% (3K hits / 5K requests)
[OK] table_definition_cache(2097152) is upper than number of tables(411)
[OK] Open file limit used: 0% (78/32K)
[OK] Table locks acquired immediately: 100% (1K immediate / 1K locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Memory used by P_S: 72B
[--] Sys schema isn't installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 8 thread(s).
[--] Using default value is good enough for your version (10.5.11-MariaDB)

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.2% (24M used / 134M cache)
[OK] Key buffer size / total MyISAM indexes: 128.0M/1.1M

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 8.5G/13.0G
[OK] Ratio InnoDB log file size / InnoDB Buffer pool size: 2.0G * 1/8.5G should be equal to 25%
[--] Number of InnoDB Buffer Pool Chunk : 68 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 96.99% (1575458 hits/ 1624411 total)
[OK] InnoDB Write log efficiency: 99.14% (279173 hits/ 281589 total)
[OK] InnoDB log waits: 0.00% (0 waits / 2416 writes)

-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/336.0K
[OK] Aria pagecache hit rate: 99.3% (2K cached / 20 reads)

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: OFF
[--] Semi synchronous replication Slave: OFF
[--] This is a standalone server

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Restrict Host for 'webappuser'@'%' to 'webappuser'@LimitedIPRangeOrLocalhost
    RENAME USER 'webappuser'@'%' TO 'webappuser'@LimitedIPRangeOrLocalhost;
    MySQL was started within the last 24 hours - recommendations may be inaccurate
    Reduce your overall MySQL memory footprint for system stability
    Dedicate this server to your database for highest performance.
    Consider installing Sys schema from https://github.com/mysql/mysql-sys for MySQL
    Consider installing Sys schema from https://github.com/FromDual/mariadb-sys for MariaDB
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    innodb_buffer_pool_size (>= 13.0G) if possible.

Answer 1

About 2013, the Optimizers in MySQL 5.6 and MariaDB 10.0 started diverging. That means that some queries run faster in one version, some in the other.

These composite indexes may help in both products:

er_reports:  (user_id, deleted_date, unique_id, name)
er_slides_tags:  (slide_id, deleted_date)
er_slides_in_report:  (deleted_date, report_id, slide_id)
er_slides_in_report:  (slide_id, report_id, deleted_date)
er_slides:   (unique_id, deleted_date, slide_order, media_id)
er_media:    (unique_id, deleted_date)

(I hope that your mixing of LEFT JOIN and INNER JOIN is not causing any trouble.)

Answer 2

I have been seeing the same thing. Up upgraded a pretty busy system to 10.5.8 and found that queries that were running in under 20 seconds on MySQL 5.6 were running over 1000 seconds on MariaDB 10.5.8. So, I upgraded to MariaDB 10.5.12, and still no joy. Looking at getting commercial support as these query time differences are untenable.