I am trying to schedule my python script in AWS, however I don't want the instances to be running all the time. So, trying to automate the process of:
I cannot run this script directly as a Lambda function because the script does some parallel processing which requires more RAM, so choosing a bigger AWS instance rather than writing it as a lambda function. Also, don't want this instance to be running all the time as it is expensive.
So far, I followed Automatic starting and stopping of AWS EC2 instances with Lambda and CloudWatch · matoski.com and created a Lambda function to start and stop the instance at specific time, however I couldn't find a way to run the python script once the instance is started.
Can anyone point me in the right direction?
MY application runs an instance @ 13:39 UST everyday and self shuts down after processing is complete. It uses below
Cloud watch Event/rules config
import boto3 def lambda_handler(event, context): ec2 = boto3.client('ec2', region_name='ap-south-1') ec2.start_instances(InstanceIds=['i-xxxxxxx']) print('started your instances: ' + str('i-xxxxxx')) return
This triggers an instance which has a cron running to execute Python script
@reboot python /home/Init.py
Once script completes, python job shuts down itself using below snippet
import boto.ec2 import boto.utils import logging logger=logging.getLogger() def stop_ec2(): conn = boto.ec2.connect_to_region("ap-south-1") # or your region # Get the current instance's id my_id = boto.utils.get_instance_metadata()['instance-id'] logger.info(' stopping EC2 :'+str(my_id)) conn.stop_instances(instance_ids=[my_id])
For future developers, who come to this question, a newer approach to this is:
AmazonEC2RoleforSSM
policySo:
Follow the steps here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
Use the following lambda skeleton:
import time
import boto3
REGION_NAME = 'us-east-1'
WORKING_DIRECTORY = '<YOUR WORKING DIRECTORY, IF ANY>'
COMMAND = """
echo "Hello, world!"
"""
INSTANCE_ID = '<YOUR INSTANCE ID>'
def start_ec2():
ec2 = boto3.client('ec2', region_name=REGION_NAME)
ec2.start_instances(InstanceIds=[INSTANCE_ID])
while True:
response = ec2.describe_instance_status(InstanceIds=[INSTANCE_ID], IncludeAllInstances=True)
state = response['InstanceStatuses'][0]['InstanceState']
print(f"Status: {state['Code']} - {state['Name']}")
# If status is 16 ('running'), then proceed, else, wait 5 seconds and try again
if state['Code'] == 16:
break
else:
time.sleep(5)
print('EC2 started')
def stop_ec2():
ec2 = boto3.client('ec2', region_name=REGION_NAME)
ec2.stop_instances(InstanceIds=[INSTANCE_ID])
while True:
response = ec2.describe_instance_status(InstanceIds=[INSTANCE_ID], IncludeAllInstances=True)
state = response['InstanceStatuses'][0]['InstanceState']
print(f"Status: {state['Code']} - {state['Name']}")
# If status is 80 ('stopped'), then proceed, else wait 5 seconds and try again
if state['Code'] == 80:
break
else:
time.sleep(5)
print('Instance stopped')
def run_command():
client = boto3.client('ssm', region_name=REGION_NAME)
time.sleep(10) # I had to wait 10 seconds to "send_command" find my instance
cmd_response = client.send_command(
InstanceIds=[INSTANCE_ID],
DocumentName='AWS-RunShellScript',
DocumentVersion="1",
TimeoutSeconds=300,
MaxConcurrency="1",
CloudWatchOutputConfig={'CloudWatchOutputEnabled': True},
Parameters={
'commands': [COMMAND],
'executionTimeout': ["300"],
'workingDirectory': [WORKING_DIRECTORY],
},
)
command_id = cmd_response['Command']['CommandId']
time.sleep(1) # Again, I had to wait 1s to get_command_invocation recognises my command_id
retcode = -1
while True:
output = client.get_command_invocation(
CommandId=command_id,
InstanceId=INSTANCE_ID,
)
# If the ResponseCode is -1, the command is still running, so wait 5 seconds and try again
retcode = output['ResponseCode']
if retcode != -1:
print('Status: ', output['Status'])
print('StdOut: ', output['StandardOutputContent'])
print('StdErr: ', output['StandardErrorContent'])
break
print('Status: ', retcode)
time.sleep(5)
print('Command finished successfully') # Actually, 0 means success, anything else means a fail, but it didn't matter to me
return retcode
def lambda_handler(event, context):
retcode = -1
try:
start_ec2()
retcode = run_command()
finally: # Independently of what happens, try to shutdown the EC2
stop_ec2()
return retcode
I was having problems starting and stopping the instance using the solutions in this post. Then I followed the instructions on https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/ and it was really easy. Basically:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"ec2:Start*",
"ec2:Stop*"
],
"Resource": "*"
}
]
}
Go to https://console.aws.amazon.com/iam/home#/home and on the left choose Roles. Make sure you choose Lambda as your AWS Service, and attach the policy you created in Step 1.
Then go to the Lambda console, click Create Function. Choose Python 3.7 and then click the dropdown next to Permissions and Use An Existing Role and attach the IAM role you created in Step 2.
Use this as your code:
import boto3
region = 'us-west-1' # Dont use the specific, like instead of us-east-1d just write us-east-1
instances = ['i-xxxxxxxxxxxx']
ec2 = boto3.client('ec2', region_name=region)
def lambda_handler(event, context):
ec2.start_instances(InstanceIds=instances)
print('started your instances: ' + str(instances))
which python
to find your path to python and write this down. Then, type in crontab -e
to edit your CRON jobs. Don't use sudo
...because sometimes sudo
messes things up when you haven't been using it to run the Python files. In my instance, I had a pgpass
file storing my password that sudo
couldn't see , but removing sudo worked!@reboot /path/to/python /path/to/file.py
For example, for me this was @reboot /home/init/python /home/init/Notebooks/mypredictor.py
import boto3
region = 'us-west-1' # Dont use the specific, like instead of us-east-1d just write us-east-1
instances = ['i-xxxxxxxxxxxx']
ec2 = boto3.client('ec2', region_name=region)
ec2.stop_instances(InstanceIds=instances)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.