Jupyter notebook behind AWS ELB terminal issue

Question

I am running Jupyter notebook on a self created docker image. This runs behind a reverse proxy (oauth2_proxy) behind an AWS ELB (that has its own dns).

Everything is working fine now, except for the Jupyter terminal.

When I connect to the instance IP directly to the Jupyter port, it works.

When I try to reach it through the ELB and the reverse-proxy it fails.

The message I see on the browser console is:

WebSocket connection to 'wss://mydomainname.domain.com/terminals/websocket/1' failed: Error during WebSocket handshake: Unexpected response code: 400

My AWS ELB is configured to accept SSL connections on 443 and redirect to TCP 8889 (where the reverse proxy is running). The reverse proxy then routes it from 8889 to 8888, where Jupyter is running.

On the jupyter config side, I already tried to add these two configs:

c.NotebookApp.trust_xheaders = True
c.NotebookApp.allow_origin = "*"

But it didnt solve the issue.

Any thoughts on how to solve this problem?

Answer 1

I encountered exactly the same issue. I finally managed to fix mine so hopefully, this can help. In my case, I don't use docker. Instead, I start jupyter lab directly in my shell and I use apache2 and proxy pass to redirect the https://mydomain:8889/jupyter to https://mydomain/jupyter . Everything works fine except the jupyter terminal, the browser console shows

WebSocket connection to 'wss://mydomain/jupyter/terminals/websocket/1' failed: Error during WebSocket handshake: Unexpected response code: 400

and the jupyter lab shows

[W 23:12:10.963 LabApp] 400 GET /jupyter/terminals/websocket/1 (2001:250:fe01:1406:605:605:605:726f) 1.22ms referer=None

The following apache2 config finally solved my problem:

RewriteEngine on
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
RewriteRule .* ws://localhost:8889%{REQUEST_URI} [P]
<Location /jupyter>
    ProxyPass http://localhost:8889/jupyter
    ProxyPassReverse http://localahost:8889/jupyter
</Location>
<LocationMatch "/jupyter/(api/kernels/[^/]+/channels|terminals/websocket)/?">
    ProxyPass ws://localhost:8889/jupyter/$1
    ProxyPassReverse ws://localhost:8889/jupyter/$1
</LocationMatch>

Jupyter terminal uses a different websocket from jupyter kernels, so LoactionMatch is needed to bypass all the websockets including the jupyter kernels, terminals and some other important websockets. Note that wss can also work here, because proxypass can bypass either ws or wss, it doesn't matter whether your website is https or http.

Also, RewriteCond is important. If you omit it, the browser console will get a "200 GET" response when you open a jupyter terminal. I don't totally understand this part, I found it in https://serverfault.com/questions/837104/apache-2-4-7-mod-proxy-wstunnel-tunneling-too-much-http-as-well-as-ws

It seems that the rewrite engine is responsible for different type of proxy header "upgrade" and "connection". I'd be glad if someone can explain why this rewrite engine is needed.

(BTW, if you are using Nginx instead of apache2, the configuration is also in the link above)

Answer 2

I fixed the issued by changing the oauth2_proxy websocket implementation.

I was having websockets issues with ipython terminal (terminado) when using grahamrhay's solution (#145) . After reverting it and adding soellman's implementation (#201) , it started working.