EC2上的Python Fabric并行执行失败：已更新

Question

因此，背景优先。 我正在运行Ubuntu 14.04，并且没有问题地运行以下脚本， put文件放在EC2实例上。 注意 ：我使用相同的IP来说明成功和失败，但是我假装我从头开始都运行了此脚本，每次都生成新的IP。

import boto.ec2
import os
from fabric.api import run, parallel, env, sudo
from fabric.tasks import execute
from fabric.operations import put

# file path python scripts and data
rps_file = "review_page_scraper.py"


# make sure hosts are clear before we add to them
env.hosts = []


# how many instances to start and how to split up the data frame 
num_instances = 3


# EC2 access keys
access_key = 'my_access_key'
secret_key = 'my_secret_key'


# get a connection to the east region
conn = boto.ec2.connect_to_region("us-east-1", 
                                  aws_access_key_id=access_key,
                                  aws_secret_access_key=secret_key)


# create the reservation of instances
reservation = conn.run_instances('my_ami_id',
                                 key_name='original_key', # my original key
                                 security_groups=['my_sec_grp'],
                                 instance_type='t2.micro',
                                 min_count=num_instances,
                                 max_count=num_instances)


# get list of instances
instance_lst = reservation.instances


# get a status update and wait if the instance isn't up and running yet
for instance in instance_lst:
    while instance.state != "running":
        sleep(5)
        instance.update()
    print "%s is running" % instance.ip_address


# get username and host, add 'ubuntu' as username
hosts = ["ubuntu@" + ip.ip_address for ip in instance_lst]
env.hosts = hosts # set environment variable


@parallel
def upload_scripts_data(file_name):
    path = "~/amazon_proj/amazon/"
    put(path + file_name, "~") # put it in the home dir of EC2 instance


# execute functions w/ rps_file
execute(upload_scripts_data, rps_file) # send review_page_scraper helpers

这是输出：

In [25]: execute(upload_scripts_data, rps_file) # send    review_page_scraper helpers
[ubuntu@52.90.34.75] Executing task 'upload_scripts_data'
[ubuntu@54.173.57.59] Executing task 'upload_scripts_data'
[ubuntu@54.165.186.168] Executing task 'upload_scripts_data'
[ubuntu@52.90.34.75] put: /home/rerwin21/amazon_proj/amazon/review_page_scraper.py -> /home/ubuntu/review_page_scraper.py
[ubuntu@54.165.186.168] put: /home/rerwin21/amazon_proj/amazon/review_page_scraper.py -> /home/ubuntu/review_page_scraper.py
[ubuntu@54.173.57.59] put: /home/rerwin21/amazon_proj/amazon/review_page_scraper.py -> /home/ubuntu/review_page_scraper.py
Out[25]: 
{u'ubuntu@52.90.34.75': None,
 u'ubuntu@54.165.186.168': None,
 u'ubuntu@54.173.57.59': None}

现在，问题是：我破坏了我的Ubuntu安装并丢失了使用ssh-keygen -t rsa生成的密钥对，当我将公钥导入AWS时将其称为“ original_key”。 因此，我必须重新安装Ubuntu，然后选择16.04。 我使用ssh-keygen -t rsa生成了一个新密钥，并将其分别保存为~/.ssh/id_rsa和~/.ssh/id_rsa和〜/ .ssh / id_rsa.pub。

然后，我导入了公共密钥，并将其保存为“ id_rsa_pub”名称。 因此，现在我运行上面的相同脚本，将key_name参数更改为“ id_rsa_pub”。 另外，我按照AWS的指示运行了chmod 0400 id_rsa 。 输出是：

In [22]: execute(upload_scripts_data, rps_file) 
[ubuntu@52.90.34.75] Executing task 'upload_scripts_data'
[ubuntu@54.173.57.59] Executing task 'upload_scripts_data'
[ubuntu@54.165.186.168] Executing task 'upload_scripts_data'
!!! Parallel execution exception under host u'ubuntu@54.165.186.168':
Process ubuntu@54.165.186.168:
Traceback (most recent call last):
 File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
 File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 242, in inner
submit(task.run(*args, **kwargs))
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
return self.wrapped(*args, **kwargs)
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/decorators.py", line 181, in inner
Traceback (most recent call last):

 File "<ipython-input-22-ed18eb00cc62>", line 1, in <module>
execute(upload_scripts_data, rps_file)

 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 412, in execute
ran_jobs = jobs.run()

 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/job_queue.py", line 168, in run
self._fill_results(results)

 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/job_queue.py", line 191, in _fill_results
datum = self._comms_queue.get_nowait()

 File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/queues.py", line 152, in get_nowait
return self.get(False)

 File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/queues.py", line 135, in get
res = self._recv()

TypeError: ('__init__() takes exactly 2 arguments (3 given)', <class 'paramiko.ssh_exception.NoValidConnectionsError'>, (None, 'Unable to connect to port 22 on  or 54.165.186.168'))

  return func(*args, **kwargs)
 File "<ipython-input-19-ed4344124d24>", line 4, in upload_scripts_data
put(path + file_name, "~")
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 677, in host_prompting_wrapper
return func(*args, **kwargs)
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/operations.py", line 345, in put
ftp = SFTP(env.host_string)
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/sftp.py", line 33, in __init__
self.ftp = connections[host_string].open_sftp()
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 159, in __getitem__
self.connect(key)
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 151, in connect
user, host, port, cache=self, seek_gateway=seek_gateway)
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 603, in connect
raise NetworkError(msg, e)
NetworkError: Low level socket error connecting to host 54.165.186.168 on port 22: Unable to connect to port 22 on  or 54.165.186.168 (tried 1 time)
[ubuntu@52.90.34.75] put: /home/rerwin21/amazon_proj/amazon/review_page_scraper.py -> /home/ubuntu/review_page_scraper.py
!!! Parallel execution exception under host u'ubuntu@54.173.57.59':
Process ubuntu@54.173.57.59:
Traceback (most recent call last):
 File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
 File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 242, in inner
submit(task.run(*args, **kwargs))
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
return self.wrapped(*args, **kwargs)
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/decorators.py", line 181, in inner
return func(*args, **kwargs)
 File "<ipython-input-19-ed4344124d24>", line 4, in upload_scripts_data
put(path + file_name, "~")
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 677, in host_prompting_wrapper
return func(*args, **kwargs)
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/operations.py", line 345, in put
ftp = SFTP(env.host_string)
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/sftp.py", line 33, in __init__
self.ftp = connections[host_string].open_sftp()
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 159, in __getitem__
self.connect(key)
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 151, in connect
user, host, port, cache=self, seek_gateway=seek_gateway)
 File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 603, in connect
raise NetworkError(msg, e)
NetworkError: Timed out trying to connect to 54.173.57.59 (tried 1 time)

对于如此冗长的问题和输出，我深表歉意。 我已经用尽了所有的知识，没有看到与我的问题完全相同的在线内容。

更新：
需要注意的几件事。 我正在使用与旧密钥对相同的安全组和AMI。 接下来，更令人困惑的是，如果我再次运行execute(upload_scripts_data, rps_file)命令，并且该命令没有错误运行。

Answer 1

更新这不能解决它。 解决此问题的唯一方法是第二次运行并行命令。

非常尴尬，但我必须发布。 如前所述，我被迫完全重新安装Ubuntu，并丢失了密钥对。 我忽略的是我的ssh_config文件。 解决问题：

sudo gedit

一旦进入gedit，请取消注释端口22：

# Port 22

至

Port 22

保存，现在我又重新启动了！