[英]Schedule to start an EC2 instance and run a python script within it
I am trying to schedule my python script in AWS, however I don't want the instances to be running all the time.我正在尝试在 AWS 中安排我的 python 脚本,但是我不希望实例一直运行。 So, trying to automate the process of:
因此,尝试自动化以下过程:
I cannot run this script directly as a Lambda function because the script does some parallel processing which requires more RAM, so choosing a bigger AWS instance rather than writing it as a lambda function.我不能直接将此脚本作为 Lambda 函数运行,因为该脚本执行一些需要更多 RAM 的并行处理,因此选择更大的 AWS 实例而不是将其编写为 lambda 函数。 Also, don't want this instance to be running all the time as it is expensive.
此外,不希望此实例一直运行,因为它很昂贵。
So far, I followed Automatic starting and stopping of AWS EC2 instances with Lambda and CloudWatch · matoski.com and created a Lambda function to start and stop the instance at specific time, however I couldn't find a way to run the python script once the instance is started.到目前为止,我使用 Lambda 和 CloudWatch · matoski.com 跟踪了 AWS EC2 实例的自动启动和停止,并创建了一个 Lambda 函数来在特定时间启动和停止实例,但是我找不到运行一次 python 脚本的方法实例已启动。
Can anyone point me in the right direction?任何人都可以指出我正确的方向吗?
MY application runs an instance @ 13:39 UST everyday and self shuts down after processing is complete.我的应用程序每天在 UST 的 13:39 运行一个实例,并在处理完成后自行关闭。 It uses below
它在下面使用
Cloud watch Event/rules config Cloud watch 事件/规则配置
import boto3 def lambda_handler(event, context): ec2 = boto3.client('ec2', region_name='ap-south-1') ec2.start_instances(InstanceIds=['i-xxxxxxx']) print('started your instances: ' + str('i-xxxxxx')) return
This triggers an instance which has a cron running to execute Python script这会触发一个运行 cron 以执行 Python 脚本的实例
@reboot python /home/Init.py @reboot python /home/Init.py
Once script completes, python job shuts down itself using below snippet脚本完成后,python 作业将使用以下代码段自行关闭
import boto.ec2 import boto.utils import logging logger=logging.getLogger() def stop_ec2(): conn = boto.ec2.connect_to_region("ap-south-1") # or your region # Get the current instance's id my_id = boto.utils.get_instance_metadata()['instance-id'] logger.info(' stopping EC2 :'+str(my_id)) conn.stop_instances(instance_ids=[my_id])
For future developers, who come to this question, a newer approach to this is:对于遇到这个问题的未来开发人员来说,一个更新的方法是:
AmazonEC2RoleforSSM
policyAmazonEC2RoleforSSM
策略的角色创建您的 EC2 So:所以:
Follow the steps here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html按照此处的步骤操作: https : //docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
Use the following lambda skeleton:使用以下 lambda 框架:
import time
import boto3
REGION_NAME = 'us-east-1'
WORKING_DIRECTORY = '<YOUR WORKING DIRECTORY, IF ANY>'
COMMAND = """
echo "Hello, world!"
"""
INSTANCE_ID = '<YOUR INSTANCE ID>'
def start_ec2():
ec2 = boto3.client('ec2', region_name=REGION_NAME)
ec2.start_instances(InstanceIds=[INSTANCE_ID])
while True:
response = ec2.describe_instance_status(InstanceIds=[INSTANCE_ID], IncludeAllInstances=True)
state = response['InstanceStatuses'][0]['InstanceState']
print(f"Status: {state['Code']} - {state['Name']}")
# If status is 16 ('running'), then proceed, else, wait 5 seconds and try again
if state['Code'] == 16:
break
else:
time.sleep(5)
print('EC2 started')
def stop_ec2():
ec2 = boto3.client('ec2', region_name=REGION_NAME)
ec2.stop_instances(InstanceIds=[INSTANCE_ID])
while True:
response = ec2.describe_instance_status(InstanceIds=[INSTANCE_ID], IncludeAllInstances=True)
state = response['InstanceStatuses'][0]['InstanceState']
print(f"Status: {state['Code']} - {state['Name']}")
# If status is 80 ('stopped'), then proceed, else wait 5 seconds and try again
if state['Code'] == 80:
break
else:
time.sleep(5)
print('Instance stopped')
def run_command():
client = boto3.client('ssm', region_name=REGION_NAME)
time.sleep(10) # I had to wait 10 seconds to "send_command" find my instance
cmd_response = client.send_command(
InstanceIds=[INSTANCE_ID],
DocumentName='AWS-RunShellScript',
DocumentVersion="1",
TimeoutSeconds=300,
MaxConcurrency="1",
CloudWatchOutputConfig={'CloudWatchOutputEnabled': True},
Parameters={
'commands': [COMMAND],
'executionTimeout': ["300"],
'workingDirectory': [WORKING_DIRECTORY],
},
)
command_id = cmd_response['Command']['CommandId']
time.sleep(1) # Again, I had to wait 1s to get_command_invocation recognises my command_id
retcode = -1
while True:
output = client.get_command_invocation(
CommandId=command_id,
InstanceId=INSTANCE_ID,
)
# If the ResponseCode is -1, the command is still running, so wait 5 seconds and try again
retcode = output['ResponseCode']
if retcode != -1:
print('Status: ', output['Status'])
print('StdOut: ', output['StandardOutputContent'])
print('StdErr: ', output['StandardErrorContent'])
break
print('Status: ', retcode)
time.sleep(5)
print('Command finished successfully') # Actually, 0 means success, anything else means a fail, but it didn't matter to me
return retcode
def lambda_handler(event, context):
retcode = -1
try:
start_ec2()
retcode = run_command()
finally: # Independently of what happens, try to shutdown the EC2
stop_ec2()
return retcode
I was having problems starting and stopping the instance using the solutions in this post.我在使用这篇文章中的解决方案启动和停止实例时遇到问题。 Then I followed the instructions on https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/ and it was really easy.
然后我按照https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/上的说明操作,这真的很容易。 Basically:
基本上:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"ec2:Start*",
"ec2:Stop*"
],
"Resource": "*"
}
]
}
Go to https://console.aws.amazon.com/iam/home#/home and on the left choose Roles.转到https://console.aws.amazon.com/iam/home#/home,然后在左侧选择角色。 Make sure you choose Lambda as your AWS Service, and attach the policy you created in Step 1.
确保选择 Lambda 作为您的 AWS 服务,并附加您在步骤 1 中创建的策略。
Then go to the Lambda console, click Create Function.然后转到 Lambda 控制台,单击创建函数。 Choose Python 3.7 and then click the dropdown next to Permissions and Use An Existing Role and attach the IAM role you created in Step 2.
选择 Python 3.7,然后单击“权限”和“使用现有角色”旁边的下拉菜单,并附加您在步骤 2 中创建的 IAM 角色。
Use this as your code:将此用作您的代码:
import boto3
region = 'us-west-1' # Dont use the specific, like instead of us-east-1d just write us-east-1
instances = ['i-xxxxxxxxxxxx']
ec2 = boto3.client('ec2', region_name=region)
def lambda_handler(event, context):
ec2.start_instances(InstanceIds=instances)
print('started your instances: ' + str(instances))
which python
to find your path to python and write this down.which python
以查找您的 python 路径并将其记下。 Then, type in crontab -e
to edit your CRON jobs.crontab -e
来编辑您的 CRON 作业。 Don't use sudo
...because sometimes sudo
messes things up when you haven't been using it to run the Python files.sudo
...因为当你没有使用它来运行 Python 文件时,有时sudo
会把事情搞砸。 In my instance, I had a pgpass
file storing my password that sudo
couldn't see , but removing sudo worked!pgpass
文件存储我的sudo
看不到的密码,但是删除 sudo 起作用了!@reboot /path/to/python /path/to/file.py
For example, for me this was @reboot /home/init/python /home/init/Notebooks/mypredictor.py
@reboot /path/to/python /path/to/file.py
例如,对我来说这是@reboot /home/init/python /home/init/Notebooks/mypredictor.py
import boto3
region = 'us-west-1' # Dont use the specific, like instead of us-east-1d just write us-east-1
instances = ['i-xxxxxxxxxxxx']
ec2 = boto3.client('ec2', region_name=region)
ec2.stop_instances(InstanceIds=instances)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.