Why did the permissions of my mongodb .sock file change automatically?

Question

Today, my MongoDB database went down after weeks of it being up. After some digging around, I realized that the permissions of my mongodb-27017.sock file were incorrect.

Running chown mongod:mongod mongodb-27017.sock resolved the issue.

My MongoDB instance was running perfectly fine for weeks . How did the permissions all of a suddenly change? How can I prevent myself from running into this issue again?

For context: I'm running an Amazon Linux 2 instance on AWS.

Answer 1

After almost one year of flawless working, one of our replica members received an error and it was related to this, mongo.sock owner changed from mongod:mongod to root:root .

I started searching that if I can find what changed the files owner but unfortunately there's no way to find it after it happened. So my search lead me to auditctl .

According to man page the description is, used to control the behavior, get status, and add or delete rules.

By setting up it like audictl -w /tmp -p rwxa -k keyname , I started waiting.

Wrote a simple shell script to notify me when audictl finds out what's changing the ownership of the file. After couple hours, I received it. With the output of the audictl, you can find information like pid , syscall and uid etc.

But the most important one is comm which tells you which program used and caused this change.

For my situation by following the audictl logs, I found out that co-worker of mine just created a cronjob that effects mongo.sock file.