MariaDB backup tool Mariabackup failing with error

Question

We've recently upgraded from MariaDB 5.5 to 10.2, and swapped from innobackupex to Mariadbackup (a fork of xtrabackup). Attempting to do a full backup always fails.

I'm running the backup with:

sudo mariabackup --backup --target-dir /mnt/database_backups/test --user backups --password REDACTED

The output of the command is as follows:

180501 11:53:30 Connecting to MySQL server host: localhost, user: backups, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
Using server version 10.2.14-MariaDB-10.2.14+maria~trusty-log
mariabackup based on MariaDB server 10.2.14-MariaDB debian-linux-gnu (x86_64)
mariabackup: uses posix_fadvise().
mariabackup: cd to /var/lib/mysql/
mariabackup: open files limit requested 0, set to 1024
mariabackup: using the following InnoDB configuration:
mariabackup:   innodb_data_home_dir = .
mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
mariabackup:   innodb_log_group_home_dir = ./
mariabackup: using O_DIRECT
2018-05-01 11:53:30 140057835345792 [Note] InnoDB: Number of pools: 1
mariabackup: Generating a list of tablespaces
2018-05-01 11:53:30 140057835345792 [Warning] InnoDB: Allocated tablespace ID 2997 for warehouse/warehouses, old maximum was 0
180501 11:53:34 >> log scanned up to (2154583932391)
180501 11:53:34 [01] Copying ./ibdata1 to /mnt/database_backups/test/ibdata1
180501 11:53:35 >> log scanned up to (2154583953963)
180501 11:53:35 [01]        ...done
180501 11:53:35 [01] Copying ./warehouse/warehouses.ibd to /mnt/database_backups/test/warehouse/warehouses.ibd
180501 11:53:35 [01]        ...done

-- MORE Copying... ...done lines

180501 12:09:59 [01] Copying ./vioadmin/amazon__product_blacklist.ibd to /mnt/database_backups/test/vioadmin/amazon__product_blacklist.ibd
180501 12:09:59 [01]        ...done
180501 12:09:59 Executing FLUSH NO_WRITE_TO_BINLOG TABLES...
180501 12:09:59 Executing FLUSH TABLES WITH READ LOCK...
180501 12:09:59 Starting to backup non-InnoDB tables and files
180501 12:09:59 [01] Copying ./warehouse/warehouse_actions_archive.frm to /mnt/database_backups/test/warehouse/warehouse_actions_archive.frm
180501 12:09:59 [01]        ...done

-- MORE Copying... ...done lines

180501 12:10:01 Finished backing up non-InnoDB tables and files
180501 12:10:01 [01] Copying aria_log_control to /mnt/database_backups/test/aria_log_control
180501 12:10:01 [01]        ...done
180501 12:10:01 [01] Copying aria_log.00000001 to /mnt/database_backups/test/aria_log.00000001
180501 12:10:01 [01]        ...done
180501 12:10:01 [00] Writing xtrabackup_binlog_info
180501 12:10:01 [00]        ...done
180501 12:10:01 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
mariabackup: The latest check point (for incremental): '2154617240383'
mariabackup: Stopping log copying thread.

2018-05-01 12:10:01 140057835345792 [Note] InnoDB: Read redo log up to LSN=2154583953920
180501 12:10:01 >> log scanned up to (2154583953920)
180501 12:10:01 Executing UNLOCK TABLES
180501 12:10:01 All tables unlocked
180501 12:10:01 [00] Copying ib_buffer_pool to /mnt/database_backups/test/ib_buffer_pool
180501 12:10:01 [00]        ...done
180501 12:10:01 Backup created in directory '/mnt/database_backups/test/'
MySQL binlog position: filename 'mariadb-bin.005485', position '28883329', GTID of the last change '0-1-241386'
180501 12:10:01 [00] Writing backup-my.cnf
180501 12:10:01 [00]        ...done
180501 12:10:01 [00] Writing xtrabackup_info
180501 12:10:01 [00]        ...done
mariabackup: Redo log (from LSN 2154583538384 to 2154583953920) was copied.
mariabackup: error: failed to copy enough redo log (LSN=2154583953920; checkpoint LSN=2154617240383).

Can someone shed some light on what the problem might be? Our data directory is approximately 94G so the database is quite large and as you can see the duration was about 17 minutes. This is similar to what we had before with innobackupex.

As you can see from the log above, part way through the backup there are some lines starting with Executing FLUSH NO_WRITE_TO_BINLOG TABLES . I'm not sure if these are ok, or a problem, but they do seem a bit odd being scattered between hundreds of Copying lines. The tables listed below there are actually all InnoDB, despite it saying Starting to backup non-InnoDB tables and files .

Thank you for your help.

Answer 1

I created Mariabackup 10.2. The redo log parsing code is somewhat different from Percona xtrabackup and also Mariabackup 10.1.

Can you share the full log, so that we can find out why exactly it fails? It would be most convenient for us if you filed a new MDEV bug at https://jira.mariadb.org/ and shared the details there, and posted the link to the issue here.

I have two hypotheses. Either the Copy_online of the redo log stops too early due to a bug, or there is a lot of InnoDB background activity that causes redo log to be written after the FLUSH TABLES WITH READ LOCK has been issued near the end of the backup.

Either way, it seems possible that the Copy_last phase is unable to copy the remaining log, because the circular redo log file was overwritten in between.

Edit: Someone else filed https://jira.mariadb.org/browse/MDEV-16367 for this issue. If you have more information, please submit it there.