MongoDB service won't start after moving location (Centos 7.1.1503)

Question

Previously mongodb was located at /var/lib/mongod . I've moved this to /home/mongod , and updated the /etc/mongod.conf file to point to this location. The permissions of the folder look correct:

drwxrwxr-x. 2 mongod mongod          6 Aug  6 10:37 admin
-rw-------. 1 mongod mongod   67108864 Aug  6 11:49 admin.0
-rw-------. 1 mongod mongod   16777216 Aug  6 11:49 admin.ns
drwxrwxr-x. 2 mongod mongod          6 Aug  6 10:40 journal
drwxrwxr-x. 2 mongod mongod          6 Aug  6 10:37 local
-rw-------. 1 mongod mongod   67108864 Aug  6 11:49 local.0
-rw-------. 1 mongod mongod   16777216 Aug  6 11:49 local.ns
-rwxr-xr-x. 1 mongod mongod          0 Aug  6 11:50 mongod.lock
-rwxrwxr-x. 1 mongod mongod          0 Aug  6 10:12 mongod.lock.bak
drwxrwxr-x. 2 mongod mongod          6 Aug  6 10:37 sphere
-rw-------. 1 mongod mongod   67108864 Aug  6 11:50 sphere.0
-rw-------. 1 mongod mongod  134217728 Aug  6 11:49 sphere.1
-rw-------. 1 mongod mongod  268435456 Aug  6 11:49 sphere.2
-rw-------. 1 mongod mongod  536870912 Aug  6 11:49 sphere.3
-rw-------. 1 mongod mongod 1073741824 Aug  6 11:50 sphere.4
-rw-------. 1 mongod mongod 2146435072 Aug  6 11:50 sphere.5
-rw-------. 1 mongod mongod   16777216 Aug  6 11:50 sphere.ns
drwxr-xr-x. 2 mongod mongod          6 Aug  6 11:50 _tmp

and when I run from the command line it works:

[user@localhost home]$ sudo -u mongod mongod --dbpath /home/mongod
[sudo] password for user: 
2015-08-06T12:04:35.761+0100 [initandlisten] MongoDB starting : pid=10235 port=27017 dbpath=/home/mongod 64-bit host=localhost.localdomain
2015-08-06T12:04:35.762+0100 [initandlisten] db version v2.6.10
2015-08-06T12:04:35.762+0100 [initandlisten] git version: 5901dbfb49d16eaef6f2c2c50fba534d23ac7f6c
2015-08-06T12:04:35.762+0100 [initandlisten] build info: Linux build18.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-08-06T12:04:35.762+0100 [initandlisten] allocator: tcmalloc
2015-08-06T12:04:35.762+0100 [initandlisten] options: { storage: { dbPath: "/home/mongod" } }
2015-08-06T12:04:35.772+0100 [initandlisten] journal dir=/home/mongod/journal
2015-08-06T12:04:35.773+0100 [initandlisten] recover : no journal files present, no recovery needed
2015-08-06T12:04:36.865+0100 [initandlisten] waiting for connections on port 27017

However running the using systemctl fails:

sudo service mongod restart
Restarting mongod (via systemctl):  Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
                                                           [FAILED]

The /etc/mongod.conf file is as follows

logpath=/var/log/mongodb/mongod.log
logappend=true

# fork and run in background
fork=true

#port=27017

dbpath=/home/mongod

# location of pidfile
pidfilepath=/var/run/mongodb/mongod.pid

# Listen to local interface only. Comment out to listen on all interfaces. 
bind_ip=127.0.0.1

The output in /var/log/mongod/mongod.log is as follows:

2015-08-06T12:08:35.402+0100 ***** SERVER RESTARTED *****
2015-08-06T12:08:35.407+0100 [initandlisten] MongoDB starting : pid=10370 port=27017 dbpath=/home/mongod 64-bit host=localhost.localdomain
2015-08-06T12:08:35.407+0100 [initandlisten] db version v2.6.10
2015-08-06T12:08:35.407+0100 [initandlisten] git version: 5901dbfb49d16eaef6f2c2c50fba534d23ac7f6c
2015-08-06T12:08:35.407+0100 [initandlisten] build info: Linux build18.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-08-06T12:08:35.407+0100 [initandlisten] allocator: tcmalloc
2015-08-06T12:08:35.407+0100 [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/home/mongod" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2015-08-06T12:08:35.407+0100 [initandlisten] exception in initAndListen std::exception: boost::filesystem::status: Permission denied: "/home/mongod/local.ns", terminating
2015-08-06T12:08:35.407+0100 [initandlisten] dbexit: 
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: going to close listening sockets...
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: going to flush diaglog...
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: going to close sockets...
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: waiting for fs preallocator...
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: lock for final commit...
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: final commit...
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: closing all files...
2015-08-06T12:08:35.407+0100 [initandlisten] closeAllFiles() finished
2015-08-06T12:08:35.407+0100 [initandlisten] dbexit: really exiting now

I've tried deleting the lock file /home/mongod/mongod.lock and running a repair with sudo -u mongod mongod --dbpath /home/mongod --repair but the error still persists. This is causing major headaches!!

Answer 1

So it turns out the problem was SElinux, and the solution is the same as this question .

It was permission related and is due to the SELinux security context which is set to enforced by default on CentOS.

sudo ausearch -m avc -ts today | audit2allow

On my machine I had some mongo related audits as follows:

#============= mongod_t ============== 
allow mongod_t home_root_t:file getattr; 
allow mongod_t user_home_dir_t:dir search; 
allow mongod_t var_lib_t:lnk_file read;
allow mongod_t default_t:file getattr;

To fix the above, you do the following:

sudo chcon -Rv --type=mongod_var_lib_t /path_to_your_mongo

Where /path_to_your_mongo is where your mongod data files are located (in my case /home/mongod but often /var/lib/mongo or /data/db .