将Apache WebSocket重定向到Tomcat：mod_proxy和mod_proxy_wstunnel

Question

I am trying to redirect traffic from Apache to Tomcat by using mod_proxy and mod_proxy_wstunnel modules. HTTP traffic is redirected without problems but I am not able to successfully redirect websocket traffic with any configuration I tried so far.

I am using Apache 2.4.28 and Tomcat 8.5.13

I must say when I use Tomcat without Apache, websockets works perfectly fine:

The Tomcat connector that works for this configuration is next:

<Connector URIEncoding="UTF-8"
        compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript"
        compression="on"
        compressionMinSize="1024"
        connectionTimeout="20000"
        noCompressionUserAgents="gozilla, traviata"
        port="443"
        protocol="org.apache.coyote.http11.Http11AprProtocol"
        SSLEnabled="true"
        scheme="https"
        secure="true">
        <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
        <SSLHostConfig>
            <Certificate certificateKeyFile="/opt/tomcat/cert/privkey.pem"
                         certificateFile="/opt/tomcat/cert/cert.pem"
                         certificateChainFile="/opt/tomcat/cert/chain.pem"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>

It's clear until this point. Now I start an Apache server in front of Tomcat and the first thing I change is the Tomcat connector like this:

<Connector URIEncoding="UTF-8"
    compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript"
    compression="on"
    compressionMinSize="1024"
    connectionTimeout="20000"
    noCompressionUserAgents="gozilla, traviata"
    port="8080"
    protocol="org.apache.coyote.http11.Http11AprProtocol">
    <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
</Connector>

In Apache I successfully load next modules (I have checked they are really loaded):

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
LoadModule ssl_module modules/mod_ssl.so

This is one of the configurations I tried in my vhosts.conf file:

<VirtualHost *:443>
    ServerName www.example.com
    ServerAdmin server@example.com
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
    CustomLog /var/log/httpd/lavnet_access.log combined
    ErrorLog /var/log/httpd/lavnet_error.log

    SSLProxyEngine on
    #websocket
    RewriteEngine on
    RewriteCond %{HTTP:Upgrade} "(?i)websocket"
    RewriteRule ^/(.*)$ wss://www.example.com/$1 [P]
    #rest
    ProxyPass "/" "http://www.example.com:8080/"
    ProxyPassReverse "/" "http://www.example.com:8080/"

    SSLEngine on
    SSLCertificateFile    /etc/letsencrypt/live/example.com/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/example.com/chain.pem
</VirtualHost>

This is the configuration with best results I have achieved but it's still not working. The log traces in lavnet_error.log seems quite good when I try to establish this connection:

[Tue Oct 10 16:46:39.014980 2017] [proxy:debug] [pid 10558:tid 47319680603904] proxy_util.c(2209): [client XX.XX.XX.109:11208] AH00944: connecting wss://www.example.com:443/rest/notify/675/fgcw02lm/websocket to www.example.com:443
[Tue Oct 10 16:46:39.016495 2017] [proxy:debug] [pid 10558:tid 47319680603904] proxy_util.c(2418): [client XX.XX.XX.109:11208] AH00947: connected /rest/notify/675/fgcw02lm/websocket to www.example.com:443
[Tue Oct 10 16:46:39.016567 2017] [proxy:debug] [pid 10558:tid 47319680603904] proxy_util.c(2887): AH02824: WSS: connection established with XX.XX.XX.109:443 (*)
[Tue Oct 10 16:46:39.016590 2017] [proxy:debug] [pid 10558:tid 47319680603904] proxy_util.c(3054): AH00962: WSS: connection complete to XX.XX.XX.109:443 (www.example.com)
[Tue Oct 10 16:46:39.016603 2017] [ssl:info] [pid 10558:tid 47319680603904] [remote 217.61.129.109:443] AH01964: Connection to child 0 established (server www.example.com:443)
[Tue Oct 10 16:46:39.026370 2017] [proxy:debug] [pid 10558:tid 47319680603904] proxy_util.c(2171): AH00943: WSS: has released connection for (*)

But this is the error that Chrome shows:

I have also tried this other configuration:

<VirtualHost *:443>
    ServerName www.example.com
    ServerAdmin server@example.com
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
    CustomLog /var/log/httpd/lavnet_access.log combined
    ErrorLog /var/log/httpd/lavnet_error.log
    SSLProxyEngine on

    ProxyPass "/rest/notify/" "wss://www.example.com:8080/rest/notify"
    ProxyPassReverse "/rest/notify/" "wss://www.example.com:8080/rest/notify"

    ProxyPass "/" "http://www.example.com:8080/"
    ProxyPassReverse "/" "http://www.example.com:8080/"

    SSLEngine on
    SSLCertificateFile    /etc/letsencrypt/live/example.com/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/example.com/chain.pem
</VirtualHost>

But in this case I get a "500 (Internal Server Error)" and also I can see next traces in lavnet_error.log:

[Tue Oct 10 17:14:14.778824 2017] [proxy:warn] [pid 11924:tid 47694559057664] [client XX.XX.XXX.189:11665] AH01144: No protocol handler was valid for the URL /rest/notify/info (scheme 'wss'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule., referer: https://www.example.com/equipment-command-panel/8287/8482

I have tried many configurations but I cannot get this to work. I hope you can help me.

Thanks.

Answer 1

After many trials I have finally solved it. Next is the working configuration in case anyone needs it:

This is the vhost.conf file of Apache:

<VirtualHost *:443>
    ServerName www.example.com
    ServerAdmin admin@example.com
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
    CustomLog /var/log/httpd/lavnet_access.log combined
    ErrorLog /var/log/httpd/lavnet_error.log

    ProxyPreserveHost On
    ProxyPass / http://www.example.com:8080/
    ProxyPassReverse / http://www.example.com:8080/
    ProxyRequests Off
    RewriteEngine on
    RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
    RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
    RewriteRule .* ws://www.example.com:8080%{REQUEST_URI} [P]

    SSLEngine on
    SSLCertificateFile    /etc/letsencrypt/live/example.com/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/example.com/chain.pem
</VirtualHost>

And this one the connector defined in server.xml in Tomcat:

<Connector URIEncoding="UTF-8"
    connectionTimeout="20000"
    port="8080"
    protocol="org.apache.coyote.http11.Http11AprProtocol">
    <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
</Connector>

Thanks.

Answer 2

The accepted solution works in single instance. But I have cluster with multiple servers and I have load balancer with mod_jk module. In this case after hours of searching I could find a solution to resolve my problem. In mod_jk the name of server appended to end of JSESSIONID like this: JSESSIONID=25B9813E079BA3543C242438FD74CA84.banana .

So I could write a rule to proxy the websocket request to the proper server with the bellow code:

SSLProxyEngine On
RewriteEngine on
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
RewriteCond %{HTTP_COOKIE} (^|;\ *)JSESSIONID=.*\.(.+?)(?=;|$) [NC]
RewriteRule .* wss://%2:8443%{REQUEST_URI} [P]

As you see in the last RewriteCond I write a regex which find the name of server in the second regex group. So I write a proxy that route the request to wss://%2:8443%{REQUEST_URI} which is the correct server.

ps: Don't forget to add server names and IP to /etc/hosts