wrong entry in limits.conf , unable to ssh to host

Question

We have VirtualBox (using vagrant) env , by mistake made an entry in /etc/security/limits.conf [with out having a root shell open:( ] and now I am unable to ssh (the connections drops immediately). Previously we had one such scenario (limits done by someone else) , was able to fix using vboxmanage guestcontrol copyto CLI and was able to overwrite limits.conf and then ssh was allowed, this time around the vboxmanage CLI also hangs

Tried to open the VM in GUI and went to console and tried few options , but could not get to single user mode.

Answer 1

Since you already tried vbox cli command and the commands hang, it means even virtualbox cannot access the system or get a shell to open.

In this case you will have to bring up a ubuntu VM and use the qemu-nbd module to fix this. The steps are given below.

Bring up a very simple ubuntu vm using hashicorp's bionic64 on the same host machine by executing the following steps.

mkdir bionic

cd bionic

vagrant box add hashicorp/bionic64

vagrant init

Open the Vagrantfile and change the config.vm.box = "base" to config.vm.box = "hashicorp/bionic64"

Also mount the folder in the host where the .vdi file for the VM is located by adding the following to the Vagrant file by adding the following line(replace the file path with the correct one corresponding to your system. Here /nbd2 will be created on the ubuntu machine and will contain the files including the .vdi file.

config.vm.synced_folder "/home/topcat/VirtualBox\ VMs/your_vm", "/nbd2"

Now do vagrant up

Once the machine boots up

vagrant ssh #to ssh as vagrant

sudo su #to become root

apt-get update #This will refresh the apt cache 

apt-get install qemu

modprobe nbd (to check if the module is loaded successfully. Will exit without any output if it is installed)

qemu-nbd -c /dev/nbd1 "/nbd2/box-disk001.vdi" - (Here change the path to whatever you gave in the config.vm.synced_folder property)

mkdir -p /mnt/vdi-boot

mount /dev/nbd1p1 /mnt/vdi-boot

cd /mnt/vdi-boot/etc/security (This folder will have all the files as it were in your VM)

touch limits.conf (if the file is already there, delete it)

chmod 644 limits.conf

chown root:root limits.conf

open the /mnt/vdi-boot/etc/security/nsswitch.conf file and check if the following three lines are present

passwd:     files
shadow:     files
group:      files
umount /mnt/vdi-boot (unmounts the mounted path)

qemu-nbd -d /dev/nbd1 (disconnects from qemu-nbd)

Exit the VM and start the VM

Open another shell and try to ssh. It should go through fine this time.