Issue developing AWS Glue ETL jobs locally using a Docker container

Question

I am using an Apple M1 Pro Mac & trying to use a Docker container to developer AWS Glue Jobs locally and not use the AWS Console. I have been working through this blog post by AWS and I have pulled amazon/aws-glue-libs:glue_libs_3.0.0_image_01 from DockerHub and run the command docker run -itd -p 8888:8888 -p 4040:4040 -v ~/.aws:/root/.aws:ro --name glue_jupyter amazon/aws-glue-libs:glue_libs_3.0.0_image_01 /home/jupyter/jupyter_start.sh . The Docker container runs but gives the warning WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested , has no output logs and http://localhost:8888 does not work. If I run the command docker exec -it glue_jupyter bash it will hang for around 20 minutes and eventually give the following error and exit with the code 127:

Traceback (most recent call last):

  File "/home/glue_user/.certs/ssl_tls_setup.py", line 87, in <module>

    automated_main(keystore_path=args.keystore, keystore_pw=args.keystorepw)

  File "/home/glue_user/.certs/ssl_tls_setup.py", line 78, in automated_main

    generate_self_signed_ssl_cert(get_default_keystore_path(), keystore_pw, DEFAULT_KEY_ALIAS)

  File "/home/glue_user/.certs/ssl_tls_setup.py", line 47, in get_default_keystore_path

    instance_dns = get_instance_dns()

  File "/home/glue_user/.certs/ssl_tls_setup.py", line 15, in get_instance_dns

    private_dns = instance_metadata['local-hostname']

  File "/usr/local/lib/python3.7/site-packages/boto/utils.py", line 275, in __getitem__

    return super(LazyLoadMetadata, self).__getitem__(key)

KeyError: 'local-hostname'

Error opening input file -nokeys

-nokeys: No such file or directory

Error opening input file -nokeys

-nokeys: No such file or directory

Error opening input file -nocerts

-nocerts: No such file or directory

starting org.apache.spark.deploy.history.HistoryServer, logging to /home/glue_user/spark/logs/spark-glue_user-org.apache.spark.deploy.history.HistoryServer-1-e08471474826.out

/usr/bin/bash: /home/jupyter/jupyter_start.sh: No such file or directory

Answer 1

Try this command

docker run -itd -p 8888:8888 -p 4040:4040 -e DISABLE_SSL="true" -v ~/.aws:/root/.aws:ro --name glue_jupyter amazon/aws-glue-libs:glue_libs_3.0.0_image_01 /home/glue_user/jupyter/jupyter_start.sh

You need to disable SSL while running docker

And to run the Jupyter notebook use this address in your browser.

http://127.0.0.1:8888/lab/tree