Tomcat启动，但未在任何端口上监听

Question

I have a tomcat service configured to run on port 8380. This site was running fine until recently. I started to see this in the mod_jk.log on apache:

[3409:140197396281344] [info] jk_open_socket::jk_connect.c (627): connect to 127.0.0.1:8309 failed (errno=111)
[3409:140197396281344] [info] ajp_connect_to_endpoint::jk_ajp_common.c (995): Failed opening socket to (127.0.0.1:8309) (errno=111)
[3409:140197396281344] [error] ajp_send_request::jk_ajp_common.c (1630): (worker1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)
[3409:140197396281344] [info] ajp_service::jk_ajp_common.c (2623): (worker1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[3409:140197396281344] [info] jk_open_socket::jk_connect.c (627): connect to 127.0.0.1:8309 failed (errno=111)
[3409:140197396281344] [info] ajp_connect_to_endpoint::jk_ajp_common.c (995): Failed opening socket to (127.0.0.1:8309) (errno=111)
[3409:140197396281344] [error] ajp_send_request::jk_ajp_common.c (1630): (worker1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)
[3409:140197396281344] [info] ajp_service::jk_ajp_common.c (2623): (worker1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)
[3409:140197396281344] [error] ajp_service::jk_ajp_common.c (2643): (worker1) connecting to tomcat failed.
[3409:140197396281344] [info] jk_handler::mod_jk.c (2788): Service error=-3 for worker=worker1

I'm able to start up the tomcat service but the port it's supposed to be listening on is not accepting connections. nmap and netstat don't show the ports as available. I've also tried telnetting to to the port just in case but as i expected there's no response. I've stopped the firewall in case it's a iptables rule issue but that too had no effect. I've googled similar issues resolved by what i tried above but none of them apply to my issues. I'm not sure what else to try at this point and so I'm turning to you all!

For this particular host the CATALINA_BASE is from a mounted nfs export. I have 2 tomcat sites hosted on this server. For the sake of testing I've disabled one from starting up since they both have same issue. i figured if i can get 1 running the other will be work as well.

server.xml:

<Server port="8105" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
  <Service name="Catalina">
    <Connector port="8380" protocol="HTTP/1.1"
               connectionTimeout="20000"
               URIEncoding="UTF-8"
               redirectPort="8443" />
    <Connector port="8309" protocol="AJP/1.3" redirectPort="8443" />
    <Engine name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />
      </Host>
      <Host name="mytomcat.site.org" appBase="webapps"
            unpackWARs="true" autoDeploy="true">
      </Host>
    </Engine>
  </Service>
</Server>

My worker.properties in apache. I have 2 workers 1 for each tomcat site.

workers.tomcat_home=/opt/tomcat
workers.java_home=/usr/local/jdk/jre
ps=/

# Define workers
worker.list=worker1
worker.list=worker2

# Set properties for worker1 - localhost
worker.worker1.type=ajp13
worker.worker1.host=127.0.0.1
worker.worker1.port=8309
worker.worker1.socket_keepalive=True
worker.worker1.socket_timeout=300

# Set properties for worker2 - localhost
worker.worker2.type=ajp13
worker.worker2.host=127.0.0.1
worker.worker2.port=8409
worker.worker2.socket_keepalive=True
worker.worker2.socket_timeout=300

Apache starts up ok and is serving static pages ok. tomcat starts ok but like i said it's not listening on any port.

Any help would be appreciated. TIA!

MORE INFO: I ran the init script with -debug in the JAVA_OPTS var. and I see the following towards the bottom of the output that scrolls out to stdout:

...
user changed to 'daemon'
User 'daemon' validated
Attempting to locate Java Home in /usr/local/jdk
Attempting to locate VM configuration file /usr/local/jdk/jre/lib/jvm.cfg
Attempting to locate VM configuration file /usr/local/jdk/lib/jvm.cfg
Attempting to locate VM configuration file /usr/local/jdk/jre/lib/amd64/jvm.cfg
Found VM configuration file at /usr/local/jdk/jre/lib/amd64/jvm.cfg
Found VM server definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/server/libjvm.so
Found VM client definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/client/libjvm.so
Checking library /usr/local/jdk/lib/amd64/client/libjvm.so
Cannot locate library for VM client (skipping)
Found VM hotspot definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/hotspot/libjvm.so
Checking library /usr/local/jdk/lib/amd64/hotspot/libjvm.so
Cannot locate library for VM hotspot (skipping)
Found VM classic definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/classic/libjvm.so
Checking library /usr/local/jdk/lib/amd64/classic/libjvm.so
Cannot locate library for VM classic (skipping)
Found VM native definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/native/libjvm.so
Checking library /usr/localjdk/lib/amd64/native/libjvm.so
Cannot locate library for VM native (skipping)
Found VM green definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/green/libjvm.so
Checking library /usr/local/jdk/lib/amd64/green/libjvm.so
Cannot locate library for VM green (skipping)
Java Home located in /usr/local/jdk
+-- DUMPING JAVA HOME STRUCTURE ------------------------
| Java Home:       "/usr/local/jdk"
| Java VM Config.: "/usr/local/jdk/jre/lib/amd64/jvm.cfg"
| Found JVMs:      1
| JVM Name:        "server"
|                  "/usr/local/jdk/jre/lib/amd64/server/libjvm.so"
+-------------------------------------------------------
Running w/ LD_LIBRARY_PATH=/usr/local/jdk/jre/lib/amd64/server:/usr/local/jdk/jre/lib/amd64
wait_child 12035
redirecting stdout to /opt/www/mytomcat/logs/catalina-daemon.out and stderr to &1

Then it goes back to the shell prompt. If it completed successfully it would have additional lines scrolling out:

get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up

I also noticed both tomcat process are running as root when i expect the child process to be running as daemon . Not sure why this is happening but it may be related to the issue.

Answer 1

Based on the debug output I started to try a bunch of different things specifically around catalina-daemon.out and catalina-daemon.pid files. As it turns out because these files are on the nfs location it causes the odd issue of the service starting but not listening. Once I moved these to the local filesystem the service started up AND it was listening on the correct port again. I also noticed the child tomcat process switched to the unprivileged user daemon. The page is reachable again.

Not entirely sure why it was working before but all of a sudden its not when i made no changes to the server. Something must have changed in the nfs export from the zfs machine that was serving up the CATALINA_BASE. Somehow this may have exposed a bug that causes this issue on the tomcat server.