Unable to install package through pip on AWS EMR Notebook

Question

I'm using a Jupyter Notebook connected to an AWS EMR Cluster. The subnets and security groups attached to all nodes of the cluster are able to connect to the internet. However, when I attempt to pip install <package> from the Notebook attached to the EMR cluster, I get the following error:

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6b4c6bd710>: Failed to establish a new connection: [Errno 101] Network is unreachable')': .... .

I have also tried to make a GET request to the internet r = requests.get('https://www.google.com') , which was also unsuccessful. I'm unable to figure out what is preventing my connectivity to the internet from my Jupyter Notebook on AWS EMR.

Answer 1

Since you are able to connect to internet with cluster, so it might be the problem with security groups of notebook itself not the cluster.

Are you using default security groups for EMR notebook like described here or are you assigning your own security group? if yes,make sure you have right outbound rules, try to add outbound rules and allow traffic

Answer 2

在我的情况下，结果证明是我创建的 sg ElasticMapReduceEditors-Livy 和 ElasticMapReduceEditors-Editor 的问题，我只是在两个 sg 中添加了一个规则，允许所有出站流量测试是否解决了它，幸运的是它解决了它，然后我删除了这两个规则，并且神秘地我的笔记本继续像往常一样工作，我认为这是 aws 的不良行为。