MySQL Inndob从非常大的数据库中删除/清除行

Question

I am having some issues with deleting data from innodb tables, from what I am reading most people are saying the only way to free up space is to export the wanted data create a new tale and import it.. this seems a very rubbish way of doing it, especially on a data which is nearly 3tbs.

The issue I am having is deleting data older then 3 months to try and free up disk space, once the data is deleted the disk space does not seem to be freed up. Is there a way to purge or permanently delete rows/data to free up disk space?

Is there a more reliable way without dropping the database and restarting the service to free up disk space.

Please could some body advise me on the best approach to handling deletion of large database.

Much appreciate your time in advanced.

Thanks :)

Answer 1

One relatively efficient approach is using database partitions and dropping old data by deleting partitions. It certainly requires more complicated maintenance, but it does work.

First, enable innodb_file_per_table so that each table (and partition) goes to its own file instead of a single huge ibdata file.

Then, create a partitioned table, having one partition per range of time (day, month, week, you pick it), which results in files of some sensible size for your data set.

create table foo(     
        tid INT(7) UNSIGNED NOT NULL,
        yearmonth INT(6) UNSIGNED NOT NULL,
        data varbinary(255) NOT NULL,
        PRIMARY KEY (tid, yearmonth) 
) engine=InnoDB
PARTITION BY RANGE(yearmonth) (
        PARTITION p201304 VALUES LESS THAN (201304),
        PARTITION p201305 VALUES LESS THAN (201305),
        PARTITION p201306 VALUES LESS THAN (201306)
);

Looking in the database data directory you'll find a file for each partition. In this example, partition 'p201304' will contain all rows having yearmonth < 201304, 'p201305' will have rows for 2013-04, 'p201306' will contain all rows for 2013-05.

In practice I have actually used an integer column containing an UNIX timestamp as the partitioning key - that way it's easier to adjust the size of the partitions as time goes by. The partition edges do not need to match any calendar boundaries, they can happen every 100000 seconds or whatever results in a sensible amount of partitions (tens of partitions) while still having small enough files with your data.

Then, set up a maintenance process which creates new partitions for new data: ALTER TABLE foo ADD PARTITION (PARTITION p201307 VALUES LESS THAN (201307)) and deletes old partitions: ALTER TABLE foo DROP PARTITION p201304 . Deletion of a large partition is almost as fast as deleting the file, and it'll actually free up disk space. Also, it won't fragment the other partitions by leaving empty space scattered inside them.

If possible, make sure your frequent queries only access one or a few partitions by specifying the partition key (yearmonth in the example above), or a range of it, in the WHERE clause - that'll make them run much faster as the database won't need to look inside all the partitions to find your data.

Answer 2

Even if you use the file_per_table option you will still have this issue. The only way to "fix" it is to rebuild individual tables:

OPTIMIZE TABLE bloated_table

Note that this will lock the table during the rebuild operation, and you must have enough free space to accommodate the new table. On some systems this is impractical.

If you're frequently deleting data, you probably need to rotate the entire table periodically. Dropping a table under InnoDB with file_per_table will liberate the disk space almost immediately. If you have one table per month, you can simply drop tables representing data from three months ago.

Is it ugly to work with these? Yes. Is there an alternative? Not really. You can try going down the table partitioning rabbit hole, but that often ends up more trouble than it's worth.