Node exporter port already in use, service is failed

Question

Node exporter is in failed state, journalctl says:

level=fatal msg="listen tcp :9100: bind: address already in use" source="node_exporter.go:114"

I tried reset-failed and restarting the service still the same issue. Then i listed processes using 9100 port and killed the process (the process was a node_exporter process) but right after i kill it whether normally or with -9 it just starts a new process as a result my 9100 port is always in use and can't start node_exporter because of it. Is there a solution for this?

Answer 1

1. Learn process-id of process, who use port 9100

   netstat -lpn | grep 9100

2. Learn parent pid of process

   ps -p PID -o ppid

3. If parent process is systemd (pid is 1), then find its service name via

   systemctl status PID

   And decide what to do with it.
   If parent process is not systemd, but something like containerd-shim , this means that this process is managed via Kubernetes Daemonset. In such case you need to decide to reuse existing process, or to change port in your node_exporter.service, you may add --web.listen-address=:9101 to ExecStart property in your service manifest and apply systemctl daemon-reload , then restart your service.

Answer 2

I had the same issue, but I found a solution:

search the process that uses 9100 port:

ss -ntpl 9100

look what PID of process and then kill him:

kill <pid>

Also, remember about node taints and tolerations