[英]Schedule to start an EC2 instance and run a python script within it
我正在尝试在 AWS 中安排我的 python 脚本,但是我不希望实例一直运行。 因此,尝试自动化以下过程:
我不能直接将此脚本作为 Lambda 函数运行,因为该脚本执行一些需要更多 RAM 的并行处理,因此选择更大的 AWS 实例而不是将其编写为 lambda 函数。 此外,不希望此实例一直运行,因为它很昂贵。
到目前为止,我使用 Lambda 和 CloudWatch · matoski.com 跟踪了 AWS EC2 实例的自动启动和停止,并创建了一个 Lambda 函数来在特定时间启动和停止实例,但是我找不到运行一次 python 脚本的方法实例已启动。
任何人都可以指出我正确的方向吗?
我的应用程序每天在 UST 的 13:39 运行一个实例,并在处理完成后自行关闭。 它在下面使用
import boto3 def lambda_handler(event, context): ec2 = boto3.client('ec2', region_name='ap-south-1') ec2.start_instances(InstanceIds=['i-xxxxxxx']) print('started your instances: ' + str('i-xxxxxx')) return
这会触发一个运行 cron 以执行 Python 脚本的实例
@reboot python /home/Init.py
脚本完成后,python 作业将使用以下代码段自行关闭
import boto.ec2 import boto.utils import logging logger=logging.getLogger() def stop_ec2(): conn = boto.ec2.connect_to_region("ap-south-1") # or your region # Get the current instance's id my_id = boto.utils.get_instance_metadata()['instance-id'] logger.info(' stopping EC2 :'+str(my_id)) conn.stop_instances(instance_ids=[my_id])
对于遇到这个问题的未来开发人员来说,一个更新的方法是:
AmazonEC2RoleforSSM
策略的角色创建您的 EC2所以:
按照此处的步骤操作: https : //docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
使用以下 lambda 框架:
import time
import boto3
REGION_NAME = 'us-east-1'
WORKING_DIRECTORY = '<YOUR WORKING DIRECTORY, IF ANY>'
COMMAND = """
echo "Hello, world!"
"""
INSTANCE_ID = '<YOUR INSTANCE ID>'
def start_ec2():
ec2 = boto3.client('ec2', region_name=REGION_NAME)
ec2.start_instances(InstanceIds=[INSTANCE_ID])
while True:
response = ec2.describe_instance_status(InstanceIds=[INSTANCE_ID], IncludeAllInstances=True)
state = response['InstanceStatuses'][0]['InstanceState']
print(f"Status: {state['Code']} - {state['Name']}")
# If status is 16 ('running'), then proceed, else, wait 5 seconds and try again
if state['Code'] == 16:
break
else:
time.sleep(5)
print('EC2 started')
def stop_ec2():
ec2 = boto3.client('ec2', region_name=REGION_NAME)
ec2.stop_instances(InstanceIds=[INSTANCE_ID])
while True:
response = ec2.describe_instance_status(InstanceIds=[INSTANCE_ID], IncludeAllInstances=True)
state = response['InstanceStatuses'][0]['InstanceState']
print(f"Status: {state['Code']} - {state['Name']}")
# If status is 80 ('stopped'), then proceed, else wait 5 seconds and try again
if state['Code'] == 80:
break
else:
time.sleep(5)
print('Instance stopped')
def run_command():
client = boto3.client('ssm', region_name=REGION_NAME)
time.sleep(10) # I had to wait 10 seconds to "send_command" find my instance
cmd_response = client.send_command(
InstanceIds=[INSTANCE_ID],
DocumentName='AWS-RunShellScript',
DocumentVersion="1",
TimeoutSeconds=300,
MaxConcurrency="1",
CloudWatchOutputConfig={'CloudWatchOutputEnabled': True},
Parameters={
'commands': [COMMAND],
'executionTimeout': ["300"],
'workingDirectory': [WORKING_DIRECTORY],
},
)
command_id = cmd_response['Command']['CommandId']
time.sleep(1) # Again, I had to wait 1s to get_command_invocation recognises my command_id
retcode = -1
while True:
output = client.get_command_invocation(
CommandId=command_id,
InstanceId=INSTANCE_ID,
)
# If the ResponseCode is -1, the command is still running, so wait 5 seconds and try again
retcode = output['ResponseCode']
if retcode != -1:
print('Status: ', output['Status'])
print('StdOut: ', output['StandardOutputContent'])
print('StdErr: ', output['StandardErrorContent'])
break
print('Status: ', retcode)
time.sleep(5)
print('Command finished successfully') # Actually, 0 means success, anything else means a fail, but it didn't matter to me
return retcode
def lambda_handler(event, context):
retcode = -1
try:
start_ec2()
retcode = run_command()
finally: # Independently of what happens, try to shutdown the EC2
stop_ec2()
return retcode
我在使用这篇文章中的解决方案启动和停止实例时遇到问题。 然后我按照https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/上的说明操作,这真的很容易。 基本上:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"ec2:Start*",
"ec2:Stop*"
],
"Resource": "*"
}
]
}
转到https://console.aws.amazon.com/iam/home#/home,然后在左侧选择角色。 确保选择 Lambda 作为您的 AWS 服务,并附加您在步骤 1 中创建的策略。
然后转到 Lambda 控制台,单击创建函数。 选择 Python 3.7,然后单击“权限”和“使用现有角色”旁边的下拉菜单,并附加您在步骤 2 中创建的 IAM 角色。
将此用作您的代码:
import boto3
region = 'us-west-1' # Dont use the specific, like instead of us-east-1d just write us-east-1
instances = ['i-xxxxxxxxxxxx']
ec2 = boto3.client('ec2', region_name=region)
def lambda_handler(event, context):
ec2.start_instances(InstanceIds=instances)
print('started your instances: ' + str(instances))
which python
以查找您的 python 路径并将其记下。 然后,输入crontab -e
来编辑您的 CRON 作业。 不要使用sudo
...因为当你没有使用它来运行 Python 文件时,有时sudo
会把事情搞砸。 在我的例子中,我有一个pgpass
文件存储我的sudo
看不到的密码,但是删除 sudo 起作用了!@reboot /path/to/python /path/to/file.py
例如,对我来说这是@reboot /home/init/python /home/init/Notebooks/mypredictor.py
import boto3
region = 'us-west-1' # Dont use the specific, like instead of us-east-1d just write us-east-1
instances = ['i-xxxxxxxxxxxx']
ec2 = boto3.client('ec2', region_name=region)
ec2.stop_instances(InstanceIds=instances)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.