eclipse: remote debugging a tomcat server behind a firewall

Question

After starting tomcat with jpda on, while at my company I can remote debug a bunch of web applications in eclipse. For a number of reasons I am now in need of developing and remote debugging those same webapps from outside the company firewall, and i can only access that server via ssh on port 22.

I tunneled most needed ports (svn, nexus, tomcat itself, from the server or via the server) to localhost and those services work fine, but I cannot start the eclipse debugger in any way; i'm getting "connection timed out while waiting for packet XXX" or "connection refused" from the second time i try on. Checking with nmap on the server, it reports the port open before the first connection attempt, and it becomes closed after that. I get no interesting output log in catalina.out

The command i use to start the tunnel is:

ssh -L 8000:localhost:8000 user@mycompany.com

iptables was temporarily stopped both on the server and in the local machine for testing.

Am i missing something? Do I need to forward some other port to localhost? Or is it in some way involved name resolution?

EDIT

Open ports before connection attemp from eclipse:

root@lnxulisse:/opt/apache-tomcat-6.0.32/bin# lsof -p 2147  -n |grep TCP
java    2147 root    4u  IPv4 640850      0t0     TCP *:8000 (LISTEN)
java    2147 root   38u  IPv6 640859      0t0     TCP *:http-alt (LISTEN)
java    2147 root   40u  IPv6 640865      0t0     TCP *:https (LISTEN)
java    2147 root   46u  IPv6 640908      0t0     TCP 127.0.0.1:18005 (LISTEN)
java    2147 root   48r  IPv6 642625      0t0     TCP 172.24.0.82:48347->172.24.0.82:mysql (ESTABLISHED)
java    2147 root  181u  IPv6 640891      0t0     TCP 172.24.0.82:60353->172.24.0.82:mysql (ESTABLISHED)

and after:

java    2147 root    4u  IPv6 642769      0t0     TCP 172.24.0.82:48956->172.24.0.82:mysql (ESTABLISHED)
java    2147 root    5u  IPv4 640851      0t0     TCP 127.0.0.1:8000->127.0.0.1:34193 (ESTABLISHED)
java    2147 root   38u  IPv6 640859      0t0     TCP *:http-alt (LISTEN)
java    2147 root   40u  IPv6 640865      0t0     TCP *:https (LISTEN)
java    2147 root   46u  IPv6 640908      0t0     TCP 127.0.0.1:18005 (LISTEN)
java    2147 root  181u  IPv6 640891      0t0     TCP 172.24.0.82:60353->172.24.0.82:mysql (ESTABLISHED)

exact eclipse error returned is:

Exception occurred during launch
Failed to connect to remote JVM. Connection timed out.
Timeout occurred while waiting for packet 204.

(the packet number varies on each attempt).

in workspace/.metadata/.log i get:

!ENTRY org.eclipse.osgi 2 0 2011-07-17 18:43:53.024
!MESSAGE While loading class "org.eclipse.core.net.proxy.IProxyService", thread "Thread[main,6,main]" timed out waiting (5000ms) for thread "Thread[Thread-6,5,main]" to finish starting bundle "org.eclipse.core.net_1.2.1.r35x_20090812-1200 [232]". To avoid deadlock, thread "Thread[main,6,main]" is proceeding but "org.eclipse.core.net.proxy.IProxyService" may not be fully initialized.
!STACK 0
org.osgi.framework.BundleException: State change in progress for bundle "reference:file:plugins/org.eclipse.core.net_1.2.1.r35x_20090812-1200.jar" by thread "Thread-6".
        at org.eclipse.osgi.framework.internal.core.AbstractBundle.beginStateChange(AbstractBundle.java:1073)
        at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:278)
[...]

!ENTRY org.eclipse.ui.ide 4 4 2011-07-17 18:43:53.028
!MESSAGE Proxy service could not be found.

eclipse is configured for direct internet connection.

EDIT 2

I think the solution might be here:

http://blog.cantremember.com/debugging-with-jconsole-jmx-ssh-tunnels/

but i have some trouble understanding his JNDI/RMI settings, and to what extent that applies to my configuration.

EDIT 3

This is a clarification for those answering "use <lan|local ip address> instead of <localhost> "

• computer A: my workstation in the company
• computer B: my workstation at home
• computer C: server running tomcat

B and C are in two different sub-networks in the same network infrastructure; only connections to port 22 of C from outside are allowed (and somewhat "proxied", I don't know network internals).

A is "outside" (my dsl connection with dynamic ip address).

Debugging on C from B via ssh tunnel -> works
Debugging on C from A via ssh tunnel -> connection timed out while waiting for packet XXX

Answer 1

This article suggests that the default port on which the remote Java virtual machine (JVM) is listening in debugging mode is 1044. You should tunnel the port on which the remote JVM is running as well.

---

More generally, you could run wireshark/tcpdump to see to which port connection attempts are made when starting the debugger.

---

EDIT :

A few more things I would try:

• check on the remote host (eg with ps auxwww if it's Linux) with which arguments (look for what comes behind -Xrunjdwp or with lsof -p PID_OF_JVM_TO_BE_DEBUGGED on which TCP port it listens (look for lines with TCP and LISTEN in the lsof output)
• make sure that the JVM on the remote host listens on the lo interface, not the network interface (that's what you specify with the localhost in the -L option to ssh).
• Does starting the debugger by hand on the machine where you start eclipse with jdb -attach localhost:8000 work ? (you could also try this on the remote host to ensure the debugger is running on the port 8000)
• make sure that eclipse tries to connect to localhost (when not specifying a bind address before the first 8000 with the -L option ssh listens on the lo interface)

Answer 2

I often had this problem when doing remote debugging. I do not know the exact reason for this problem, but I used the below solution and maybe it works for you, too:

instead of

ssh -L 8000:localhost:8000 user@remotehost

is used

ssh -L 8000:remotehost:8000 user@remotehost

for creating the SSH tunnel (note the remotehost instead of localhost between the port numbers in the second example). Instead of the remote host's name, you can also use the normal IP address of the remote host (not the loopback address 127.0.0.1, but the true local network IP address).

Hope it helps and good luck!

Answer 3

Assuming the remote Tomcat instance has been started with something like -Xrunjdwp:transport=dt_socket,server=y,address=8000,suspend=n , try this command:

ssh -L 8000:0.0.0.0:8000 user@mycompany.com -N

On my Mac, I tried out ssh -L 10701: :10700 user@localhost -N locally, where a Tomcat instance was started with -Xrunjdwp:transport=dt_socket,server=y,address=10700,suspend=n , and attempting to attach on port 10701 within Eclipse, I kept seeing "Failed to connect to remote VM com.sun.jdi.connect.spi.ClosedConnectionException". 尝试了ssh -L 10701: :10700 user@localhost -N ，其中使用-Xrunjdwp:transport=dt_socket,server=y,address=10700,suspend=n启动Tomcat实例-Xrunjdwp:transport=dt_socket,server=y,address=10700,suspend=n ，并尝试要连接到Eclipse中的端口10701，我一直看到“无法连接到远程VM com.sun.jdi.connect.spi.ClosedConnectionException”。 By changing the tunnel command to ssh -L 10701: :10700 user@localhost -N , Eclipse was able to attach. :10700 user@localhost -N ，Eclipse能够附加。

Answer 4

Can you please give the exact parameters of the -Xrunjdwp parameter?

Also do you have tried different methods for debugging (server=y/n, suspend=y/n)?

Perhaps inversing the connection (let the tomcat connect to the debugger instead of letting the debugger connect to tomcat) may help.

Answer 5

Well I am answering myself after a long time; in my specific case, the solution was to put eclipse JVM in listening mode:

Connection Type: "Standard (Socket Listen)"

and reverse the direction of the tunnel:

ssh -L 8001:localhost:8001 user@work   (run on server (S), "localhost" is W)
ssh -R 8001:localhost:8001 user@work   (run at home (H), "localhost" is W)

Some explanation: as in the question, my situation was:

  H  -------------------> S     not working  ( ssh -L 8001:S:8001 user@S  from H)
  H           W  -------> S     working      ( ssh -L 8001:S:8001 user@S  from W)
 home        work      server

While reversing like this:

  H  <------- W           S     ssh -R 8001:localhost:8001 user@W  (from H)
  H           W  <------- S     ssh -L 8001:localhost:8001 user@W  (from S)
 home        work      server

did the trick. In other words, whatever is written on S:8001, is forwarded to W:8001, and whatever in turn is written to W:8001, is forwarded to H:8001, where my eclipse JVM is listening.

The tomcat JVM on S should be started with server=n, with arguments:

-agentlib:jdwp=transport=dt_socket,server=n,suspend=n,address=8001