AWS EMR JupyterHub 設置了 LDAP 但沒有啟動筆記本

Question

我已經啟動了一個包含 jupyterhub 的 EMR 集群，並按照以下指南設置了 LDAP：

https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-jupyterhub-ldap-users.html

我可以使用 LDAP 登錄 jupyterhub，但它不會啟動筆記本服務器並引發錯誤：

500 : Internal Server Error

Spawner failed to start [status=1]. The logs for joe.blogg may contain 
details.

You can try restarting your server from the home page.

檢查 jupyterhub 日志我發現以下錯誤：

[I 2019-04-30 12:24:11.590 JupyterHub log:122] 302 GET / → /hub (@::ffff:172.31.150.206) 0.75ms
[I 2019-04-30 12:24:11.629 JupyterHub log:122] 302 GET /hub → /hub/home (joe.blogg@::ffff:172.31.150.206) 2.22ms
[I 2019-04-30 12:24:11.685 JupyterHub log:122] 200 GET /hub/home (joe.blogg@::ffff:172.31.150.206) 26.51ms
[I 2019-04-30 12:24:13.741 JupyterHub log:122] 302 GET /hub/spawn → /user/joe.blogg/ (joe.blogg@::ffff:172.31.150.206) 3.70ms
[I 2019-04-30 12:24:13.769 JupyterHub log:122] 302 GET /user/joe.blogg/ → /hub/user/joe.blogg/ (@::ffff:172.31.150.206) 0.51ms
[I 2019-04-30 12:24:13.847 JupyterHub spawner:978] Spawning jupyterhub-singleuser --port=46821 --debug
[D 2019-04-30 12:24:14.255 SingleUserNotebookApp application:177] Searching ['/home', '/home/users/joe.blogg/.jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2019-04-30 12:24:14.256 SingleUserNotebookApp application:555] Looking for jupyter_config in /etc/jupyter
[D 2019-04-30 12:24:14.256 SingleUserNotebookApp application:555] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2019-04-30 12:24:14.256 SingleUserNotebookApp application:555] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2019-04-30 12:24:14.256 SingleUserNotebookApp application:555] Looking for jupyter_config in /home/users/joe.blogg/.jupyter
[D 2019-04-30 12:24:14.256 SingleUserNotebookApp application:555] Looking for jupyter_config in /home
[D 2019-04-30 12:24:14.257 SingleUserNotebookApp application:555] Looking for jupyter_notebook_config in /etc/jupyter
[D 2019-04-30 12:24:14.257 SingleUserNotebookApp application:577] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2019-04-30 12:24:14.258 SingleUserNotebookApp application:555] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2019-04-30 12:24:14.258 SingleUserNotebookApp application:555] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2019-04-30 12:24:14.258 SingleUserNotebookApp application:577] Loaded config file: /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2019-04-30 12:24:14.258 SingleUserNotebookApp application:555] Looking for jupyter_notebook_config in /home/users/joe.blogg/.jupyter
[D 2019-04-30 12:24:14.258 SingleUserNotebookApp application:555] Looking for jupyter_notebook_config in /home
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'runtime_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 6, in <module>
    main()
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 455, in main
    return SingleUserNotebookApp.launch_instance(argv)
  File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1366, in initialize
    self.init_configurables()
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1100, in init_configurables
    connection_dir=self.runtime_dir,
  File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
    ensure_dir_exists(rd, mode=0o700)
  File "/opt/conda/lib/python3.6/site-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  [Previous line repeated 1 more times]
  File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/users'
[W 2019-04-30 12:24:23.874 JupyterHub web:1588] 500 GET /hub/user/joe.blogg/ (::ffff:172.31.150.206): Spawner failed to start [status=1]. The logs for joe.blogg may contain details.
[E 2019-04-30 12:24:23.886 JupyterHub log:114] {
      "X-Forwarded-Host": "ip-172-31-150-206.eu-west-1.compute.internal:9443",
      "X-Forwarded-Proto": "https",
      "X-Forwarded-Port": "9443",
      "X-Forwarded-For": "::ffff:172.31.150.206",
      "Upgrade-Insecure-Requests": "1",
      "Cookie": "jupyter-hub-token=\"2|1:0|10:1556624768|17:jupyter-hub-token|44:ZThlOTZkYWM0NzRiNDRkMDlmYzdkNDUwOTUzMTNjYjA=|12a53077b8d92723bba01fc9273eb64050911e22317385f96c1c4f52ff5253a8\"; _xsrf=2|5370034d|137cc417d37f89a6aed65c0ec72ad572|1556623914",
      "Connection": "close",
      "Referer": "https://ip-172-31-150-206.eu-west-1.compute.internal:9443/hub/home",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0",
      "Host": "ip-172-31-150-206.eu-west-1.compute.internal:9443"
    }
[E 2019-04-30 12:24:23.886 JupyterHub log:122] 500 GET /hub/user/joe.blogg/ (joe.blogg@::ffff:172.31.150.206) 10088.04ms
[W 2019-04-30 12:24:42.370 JupyterHub user:458] joe.blogg's server never showed up at http://127.0.0.1:46821/user/joe.blogg/ after 30 seconds. Giving up
[E 2019-04-30 12:24:42.380 JupyterHub gen:914] Exception in Future <tornado.concurrent.Future object at 0x7f53b1b9ce10> after timeout
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 910, in error_callback
        future.result()
      File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 445, in finish_user_spawn
        yield spawn_future
      File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 476, in spawn
        raise e
      File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 450, in spawn
        resp = yield server.wait_up(http=True, timeout=spawner.http_timeout)
      File "/opt/conda/lib/python3.6/site-packages/jupyterhub/utils.py", line 180, in wait_for_http_server
        timeout=timeout
      File "/opt/conda/lib/python3.6/site-packages/jupyterhub/utils.py", line 135, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at http://127.0.0.1:46821/user/joe.blogg/ didn't respond in 30 seconds

在遵循 aws 指南后，/etc/jupyter/conf/jupyterhub_conf 文件如下所示：

# Configuration file for jupyterhub.

import os
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR')
network_name='jupyterhub-network'

c.Spawner.debug = True
c.Spawner.environment = {'SPARKMAGIC_CONF_DIR':'/etc/jupyter/conf'}

c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.admin_access = True
c.JupyterHub.ssl_key = '/etc/jupyter/conf/server.key'
c.JupyterHub.ssl_cert = '/etc/jupyter/conf/server.crt'
c.JupyterHub.port = 9443


c.Authenticator.admin_users = {'jovyan'}

c.JupyterHub.authenticator_class = 'ldapauthenticator.LDAPAuthenticator'
c.LDAPAuthenticator.use_ssl = False
c.LDAPAuthenticator.server_address = 'openldap.companyx.com'
c.LDAPAuthenticator.bind_dn_template = 'cn={username},ou=Users,dc=openldap,dc=companyx,dc=com'

請問有沒有其他人設法在 AWS EMR jupyterHub 上設置多用戶訪問。

Answer 1

我遇到過類似的問題，你能試着評論一下嗎

#c.JupyterHub.ssl_key = '/etc/jupyter/conf/server.key'
#c.JupyterHub.ssl_cert = '/etc/jupyter/conf/server.crt'

在嘗試使用 http（而不是 https）訪問之后，這應該可以解決您的問題（希望沒有 https 對您來說沒問題）