How to remotely launch a Juypter Notebook within AWS Sagemaker using AWS Lambda

Question

I have a Juypter Notebook set up within AWS Sagemaker. I wanted to find a way to launch this notebook on an autonomous trigger when a new file is uploaded to a certain folder (hence AWS Lambda). I was looking for if there was a streamlined way to trigger a Juypter Notebook with an AWS Lambda trigger.

I have looked into using API and turning Sagemaker into and endpoint, but it didnt work.

*edit Sorry if the question was a little vague. I have allot of code written in this notebook on in Juypter. What i was ideally looking for was, when a file is uploaded to "RandomFile" then the code within the notebook will run. I was looking to do this with AWS Lambda by setting up a S3 based trigger.

Answer 1

Thanks for using Amazon SageMaker! Very interesting use case!

Currently, there isn't any direct way to trigger the execution of a Jupyter Notebook upon an S3-based trigger, but there are a few ways you might be able to accomplish this:

SQS-based solution

1. Configure your S3 bucket to send Event Notifications to send messages to an SQS queue .
2. Configure a cron job on your SageMaker Notebook Instance that continually polls the queue for work.
   • Example shell-script based SQS poller - any language of course would work.
   • Example configuration of a cron job in a Lifecycle Configuration
3. When a message is detected by the poller, execute nbconvert or papermill (as suggested by Guy) on your notebook.
4. Configure a CloudWatch Alarm on the size of your SQS Queue, so you are alerted if your cron job is no longer working.

Then, just leave the Notebook Instance up-and-running at all times, and it should detect when objects are uploaded to S3.

Lambda and Step Functions Solution

This solution is a bit more complex, but has the advantage of being more cost-effective if you don't want your Notebook Instance running at all times.

1. Configure your Notebook Instance with a Lifecycle Configuration which does the following:
   1. Pull the file from S3
   2. Run nbconvert in the background.
   3. Upload the result of nbconvert to S3.
2. Configure your S3 bucket to sent Event notifications to Amazon Step Functions.
3. Create a Step Functions State Machine which has the following steps:
   1. Start your notebook instance
   2. Wait for the result of nbconvert to be uploaded to S3.
   3. Stop the notebook instance.

Other Solutions

• Use the nbconvert RESTful API. The nbconvert API is available at /api/nbconvert, however it's not well-documented from what I can tell. To do this in a Lambda function, you can get a Presigned URL to the Notebook Instance and then interact with this API via the Python Requests library (or similar). I haven't tested this, however, so YMMV.