简体繁体中英

store data in one large table or create new table every month

原文 2016-02-29 01:27:48 2 1 mysql/ sql/ performance/ mariadb

I'm designing a table in mysql that storing thousand of devices and about hundred of records per device every 10 seconds. This table is aimed to log device data for analysis, some sort of historian or time series database but on transactional database.. It has 6 columns: ID (PK), Timestamp, TagID, DeviceID, Decimal Value, MessageID. I'm indexing Timestamp, TagID, DeviceID, Decimal Value. Do we have any potential problem when this table grow significantly large in term of dba, select query? Are we better off to create new table every month to keep the table size small? I'm not a database expert so I don't know which way are better. Would anyone please advice. Many thanks.

1 answers

What will the SELECTs look like? If you always want all the data for a given device&time, then pack them into a single column. I suggest JSON -encoding. Otherwise, you are asking for INSERTing 10,000 rows per second?

If you do need to keep the data items in separate rows, the be sure to use some form of batching -- multi-row INSERT (100 rows at a time), or LOAD DATA from a file.

When the table gets to be huge, there are many possible problems. (Again, I need to see the SELECTs to go into more detail.)

PARTITIONIng ? Not likely to help. Especially not BY HASH . Multiple tables are not likely to help. (Again, I need to see the SELECTs to go into more detail.)

"I'm indexing Timestamp, TagID, DeviceID, Decimal Value" -- Do you mean one-column indexes? Likely to be very much a waste. (Again, I need to see the SELECTs to go into more detail.)

If you need to do large SELECTs to summarize the data; then let's talk about Summary tables.

Please provide SHOW CREATE TABLE . You need to shrink the datatypes to save space.

Which way is legitimate to store &process big amounts of data in database? Store all in one table or create new table by id (property)for each?

Create a new table or store in an array

Mysql select from multiple tables and a new table is created every month

Want to have a table where a new row is added every month

MYSQL and PHP calculate total price of orders table every day and every month, populate new summary table

Database Schema - Create one large table or many?

Create table per month

Create a new table with a column for every element in another table

Trigger calling a store procecure is updating every data on table, instead of only one

Better to store multiple values in a table column, or create new table?

暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question Which way is legitimate to store &process big amounts of data in database? Store all in one table or create new table by id (property)for each? Create a new table or store in an array Mysql select from multiple tables and a new table is created every month Want to have a table where a new row is added every month MYSQL and PHP calculate total price of orders table every day and every month, populate new summary table Database Schema - Create one large table or many? Create table per month Create a new table with a column for every element in another table Trigger calling a store procecure is updating every data on table, instead of only one Better to store multiple values in a table column, or create new table?

Related Tags

store data in one large table or create new table every month

Question

1 answers

solution1 0 2016-02-29 23:09:27

solution1
0 2016-02-29 23:09:27