Free up space in MySQL 5.6.20 - InnoDB

Question

first off, not a DB guy. here is the problem, data drive for the database is 96% full. in the my.cnf there is a line that has the following, (only showing part due to space)

innodb_data_file_path=nmsdata1:4000M;nmsdata2:4000M; 

going up to

nmsdata18:4000M:autoextend

so in the folder where the files are stored files 1-17 are 4gb in size, file 18 is 136gb as of today.

I inherited the system and it has no vendor support or much documentation. I can see there are a few tables that are really large

Table_name                             NumRows              Data Length
----------                             -------              -----------
pmdata                                 100964536            14199980032
fault                                  310864227            63437946880
event                                  385910821            107896160256

I know ther is a ton of writes happening and there should be a cron job that tells it to only keep the last 3 months data but I am concerned the DB is fragmented and not releasing space back for use.

so my task is to free up space in the DB so the drive does not fill up.

Answer 1

This is a weakness of innodb: tablespaces never shrink. They grow, and even if you "defragment" the tables, they just get written internally to another part of the tablespace, leaving more of the tablespace "free" for use by other data, but the size of the file on disk does not shrink.

Even if you DROP TABLE, that doesn't free space to the drive.

This has been a sore point for InnoDB for a long time: https://bugs.mysql.com/bug.php?id=1341 (reported circa 2003).

The workaround is to use innodb_file_per_table=1 in your configuration, so each table has its own tablespace. Then when you use OPTIMIZE TABLE <tablename> it defragments by copying data to a new tablespace, in a more efficient, compact internal layout, and then drops the fragmented one.

But there's a big problem with this in your case. Even if you were to optimize tables after setting innodb_file_per_table=1 , their data would be copied into new tablespaces, but that still wouldn't shrink or drop the old multi-table tablespaces like your nmsdata1 through 18. They would still be huge, but "empty."

What I'm saying is that you're screwed. There is no way to shrink these tablespaces, and since you're full up on disk space, there's no way to refactor them either.

Here's what I would do: Build a new MySQL Server. Make sure innodb_file_per_table=1 is configured. Also configure the default for the data file path: innodb_data_file_path=ibdata1:12M:autoextend . That will make the central tablespace small from the start. We'll avoid expanding it with data.

Then export a dump of your current database server, all of it. Import that into your new MySQL server. It will obey the file-per-table setting, and data will create and fill new tablespaces, one per table.

This is also an opportunity to build the new server with larger storage, given what you know about the data growth.

It will take a long time to import so much data. How long depends on your server performance specifications, but it will take many hours at least. Perhaps days. This is a problem if your original database is still taking traffic while you're importing.

The solution to that is to use replication, so your new server can "catch up" from the point where you created the dump to the current state of the database. This procedure is documented, but it may be quite a bit of learning curve for someone who is not a database pro, as you said: https://dev.mysql.com/doc/refman/8.0/en/replication-howto.html

You should probably get a consultant who knows how to do this work.