MariaDB 10.1.13 on ubuntu trusty: no other datadir possible

Question

On Ubuntu Trusty Server 64bit:

I had MariaDB 10.1 with a different datadir working

datadir=/mnt/SRVDATA/var/lib/mysql

Yesterday I got an update 10.1.13-MariaDB-1~trusty from

deb http://mirror.netcologne.de/mariadb/repo/10.1/ubuntu trusty main

And MariaDB won't start any more.

I renamed /etc/mysql, /usr/lib/mysql and /var/lib/mysql and I reinstalled MariaDB Server and Client. I made a dpkg-reconfigure and I got a new pristine database in /var/lib/mysql. With it I could start MariaDB.

I copied /var/lib/mysql again to /mnt/SRVDATA/var/lib/mysql with all permissions, I changed my.cnf to point datadir to /mnt/SRVDATA/var/lib/mysql but MariaDB won't start.

Apparmor is down, it's profiles are teardown.

The errors in syslog:

Mar 30 09:58:47 hg-bioinfo mysqld: 
Mar 30 09:58:47 hg-bioinfo mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Mar 30 10:08:04 hg-bioinfo mysqld_safe: Starting mysqld daemon with databases from /mnt/SRVDATA/var/lib/mysql
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [Note] /usr/sbin/mysqld (mysqld 10.1.13-MariaDB-1~trusty) starting as process 30831 ...
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [Note] InnoDB: The InnoDB memory heap is disabled
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [Note] InnoDB: Memory barrier is not used
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [Note] InnoDB: Compressed tables use zlib 1.2.8
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [Note] InnoDB: Using Linux native AIO
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [Note] InnoDB: Using SSE crc32 instructions
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [Note] InnoDB: Initializing buffer pool, size = 256.0M
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [Note] InnoDB: Completed initialization of buffer pool
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [Note] InnoDB: Highest supported file format is Barracuda.
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [ERROR] InnoDB: Tried to read 512 bytes at offset 512. Was only able to read 0.
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 7fe86f43d7c0  InnoDB: Operating system error number 22 in a file operation.
Mar 30 10:08:04 hg-bioinfo mysqld: InnoDB: Error number 22 means 'Invalid argument'.
Mar 30 10:08:04 hg-bioinfo mysqld: InnoDB: Some operating system error numbers are described at
Mar 30 10:08:04 hg-bioinfo mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
Mar 30 10:08:04 hg-bioinfo mysqld:  InnoDB: Operation read to file /home/buildbot/buildbot/build/mariadb-10.1.13/storage/xtradb/os/os0file.cc and at line 3158
Mar 30 10:08:04 hg-bioinfo mysqld: 2016-03-30 10:08:04 140636275857344 [ERROR] InnoDB: File (unknown): 'read' returned OS error 222. Cannot continue operation
Mar 30 10:08:04 hg-bioinfo mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Mar 30 10:09:05 hg-bioinfo /etc/init.d/mysql[31401]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Mar 30 10:09:05 hg-bioinfo /etc/init.d/mysql[31401]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
Mar 30 10:09:05 hg-bioinfo /etc/init.d/mysql[31401]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")'
Mar 30 10:09:05 hg-bioinfo /etc/init.d/mysql[31401]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!

Deeply grateful for any hints!

Best regards

Answer 1

After having a great chat with Montywi on FreeNode in #maria it turns out, that this is a new bug introduced in the version 10.1.13.

It concerns the configuration:

innodb_flush_method=O_DIRECT

in combination with a datadir on a file system with a block size <> 512 (mine is on a RAID)

The workaround for now is:

innodb_flush_method=O_DSYNC

The bug is beeing fixed now and will be available soon in version 10.1.14, so Montywi.

Thank you folks!