MYSQL InnoDB:why performance after increasing the buffer pool size isn't even close to the MEMORY engine?

Question

I have a database containing a single table. the size of the table is 3.5 Gs.

I'm doing a read only query on the table, using three different configurations:
1- Innodb default buffer pool size.
2- Innodb buffer pool size = 6G.
3- Memory engine.

the running times of the three different configurations:
1- default buffer pool size .... 15,53 seconds.
2- buffer pool size = 6G ...... 13,60 seconds.
3- Memory engine .... 3,96 sec.
....

if increasing the buffer pool size shall make the database like "in-memory" databases.... why there is a huge gap between Memory engine and the buffer pool with huge enough space to contain the tables.

Notes:
1- I'm doing the experiment on a dedicated machine.

2- when using the buffer pool with 6Gs.... no swapping occurs, so the table fits comfortably within the memory..without swapping.

3- I was doing the query more than once to ensure the "hot data" was loaded to the main memory... and I was watching the memory consumption... it went from 500 MB to arround 4G after doing the query .... buffer pool 6G setting.

4- the table created using this command:

CREATE TABLE lineitem ( 
L_ORDERKEY    INTEGER NOT NULL,
L_PARTKEY     INTEGER NOT NULL,
L_SUPPKEY     INTEGER NOT NULL,
L_LINENUMBER  INTEGER NOT NULL,
L_QUANTITY    DECIMAL(15,2) NOT NULL,
L_EXTENDEDPRICE  DECIMAL(15,2) NOT NULL,
L_DISCOUNT    DECIMAL(15,2) NOT NULL,
L_TAX         DECIMAL(15,2) NOT NULL,
L_RETURNFLAG  CHAR(1) NOT NULL,
L_LINESTATUS  CHAR(1) NOT NULL,
L_SHIPDATE    DATE NOT NULL,
L_COMMITDATE  DATE NOT NULL,
L_RECEIPTDATE DATE NOT NULL,
L_SHIPINSTRUCT CHAR(25) NOT NULL,
L_SHIPMODE     CHAR(10) NOT NULL,
L_COMMENT VARCHAR(44) NOT NULL);

5- the query I'm running, (ie), query 6 of the tpch

select
sum(l_extendedprice * l_discount) as revenue
from
  tpch2.lineitem
where
   l_shipdate >= date '1994-01-01'
   and l_shipdate < date '1994-01-01' + interval '1' year
   and l_discount between 0.06 - 0.01 and 0.06 + 0.01
   and l_quantity < 24;

Answer 1

• Are there no indexes? Or does the table have INDEX(l_shipdate) and INDEX(l_discount) and INDEX(l_quantity) so that the Optimizer can pick among them?
• Please provide EXPLAIN SELECT ... for both the InnoDB and Memory versions.
• Are you running one connection doing that query repeatedly? Or many? Or so many that you are maxing out resources?

INDEX(l_shipdate, l_discount, l_quantity) is not beneficial since the Optimizer can't really handle more than one "range", and each part of the WHERE is a "range".

I'm surprised that the speed ratio is more than 3:1. Memory would have to do a table scan, testing every row. InnoDB, with the 3 indexes I suggest would probably use an index. This depends on the distribution of the data. Speaking of which, how many rows in that date range? In that discount range? In that quantity range?

Did you run each timing twice? The first time would have I/O, but "warm up the cache"; the second would (presumably) have no I/O.