使用mariadb存储未来行情数据（大量记录）

Question

for future market data, we need at least 1,000,000 records each day, each record has less them 10 fileds with a few letters. i chose mariadb 5.5 on centos 7. engine is innodb. my.cnf has following configure:

[server]
innodb_file_per_table=1
innodb_flush_log_at_trx_commit=2
innodb_buffer_pool_size=2G
innodb_log_file_size=256M
innodb_log_buffer_size=8M
bulk_insert_buffer_size=256M

when i insert records, it is not so fast, but it can be accepted. but when I do export data, it is very slow when innodb talbe large than some GB. fields like: id, bid, ask, time, xx,xx, id is auto increment, and is the key. my query sql like following: select * from table where instrument="xx" and time >= "xx" and time <= "xx"

any advise how to speed up select performance? thanks!

Answer 1

To tailor to the SELECT , make the table InnoDB and set up the clustered PRIMARY KEY so that the desired rows are consecutive. This is likely to slow down the INSERT process, but that is not an issue -- 12 inserts/second is easily handled.

But let me digress for a moment -- do the 1M rows come in all at once? Or are the trickling in over 7 hours? Or what? If all at once, sort the data according to the PK before doing the massive LOAD DATA .

Your query begs for PRIMARY KEY(instrument, time) . But a PK must be "unique"; is that unique? If not, then another column ( id ?)` should be tacked onto the end to make it unique.

Note that if it is unique, then you don't need an AUTO_INCREMENT ; get rid of it. For such large tables, minimizing the number of indexes is critical, not just for performance, but for even being able to survive.

Other things to do...

• Normalize the instrument. That have a table of such and map it to an id, probably SMALLINT UNSIGNED (2 bytes) if there are under 65K. See my blog for more discussion of normalizing as you ingest.
• Shrink any fields you can -- FLOAT (4 bytes) is tempting, but it has round off errors. DECIMAL is tricky because you need to worry about penny-stocks at one extreme and BRK-A at the other.
• Look at the rest of the queries to make sure this change in PK does not hurt them.
• Set innodb_buffer_pool_size to about 70% of available RAM (assuming you have more than 4GB of RAM).
• If you do have to keep id as an AUTO_INCREMENT , then add INDEX(id) ; that is all that is needed to keep A_I happy.
• Use CHARACTER SET ascii unless you need utf8 somewhere.
• Volume can exceed 4 billion in rare cases; ponder what to do.
• Fetching 10K rows in PK order will take only seconds.
• FULLTEXT is not useful for this application.
• PARTITIONing is not likely to be useful; we can revisit it if you care to share the rest of the queries. On the other hand, if you will be deleting 'old' data, then PARTITIONing is an excellent idea. See my partition blog .