After Java OutOfMemoryError, Tomcat Won't Start

Question

After running my App in Tomcat many times, I've maxed out the PermGen memory (a problem with my quartz servlet, I suspect) and Java threw a OutOfMemoryError. Normally, I just restart tomcat from time to time, but since this happened I'm unable to start/restart. Tomcat is running in our qa environment and is normally started/stoped/restarted via the sbin folder. I've tried to start tomcat using the alternative, catalina.sh run, but it says the port is being used! That led me to suspect that tomcat was already running, but when I try to stop it, I get this error:

SEVERE: Could not contact localhost:8005. Tomcat may not be running. Jan 19, 2014 3:10:58 PM org.apache.catalina.startup.Catalina stopServer SEVERE: Catalina.stop: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) ...

So, if it's not started or stopped, then what state is tomcat in?! How can I bring it back to a startable state?

Update As per Boris's comment, I've used ps auxfww to list processes and the only one with tomcat as the user is the following:

tomcat 5111 8.0 29.0 2052336 556512 ? Sl Jan17 221:48 /usr/java/latest/bin/java -Djava.util.logging.config.file=/var/lib/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -Xmx512m -Djava.awt.headless=true -Djava.endorsed.dirs=/usr/share/tomcat/endorsed -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.io.tmpdir=/var/lib/tomcat/temp org.apache.catalina.startup.Bootstrap start

For some reason, even as root, I can't delete the process. I use

sudo kill 5111

but the process is still listed after and tomcat still won't restart...

Answer 1

Read about kill here

sudo kill -9 5111 will do the work. Tomcat is binded to 8005 port: until you kill the process with a SIGKILL signal it won't release the port (and you are not allowed to start another tomcat on this port).