AWS Glue Pyspark，结束有条件的工作？

Question

Seems like a simple task, but im having trouble finding the docs to see if it's possible. Basically I have a glue job that runs every hour and searches a folder to see if data has been uploaded. On some occasions, no data has been uploaded in the past hour, so when the Glue function runs and sees there's no data, I'd like it to terminate. Is that possible? Here's some pseudocode to illustrate what I mean:

def fn(input):
    *fetches list of data*
    return (list of data)

list_of_data = fn(input)
if list_of_data is None:
    Terminate Job

Answer 1

Yes as bdcloud mentioned it correctly, we can directly trigger the Glue job from Lambda. Have the event trigger created on the landing folder and when a file is uploaded, trigger the glue job. Please find herewith a code snippet for AWS Lambda:

from __future__ import print_function
import json
import boto3
import time
import sys
import time
from datetime import datetime

s3 = boto3.client('s3')
glue = boto3.client('glue')

def lambda_handler(event, context):
    gluejobname="<< THE GLUE JOB NAME >>"

    try:
        runId = glue.start_job_run(JobName=gluejobname)
        status = glue.get_job_run(JobName=gluejobname, RunId=runId['JobRunId'])
        print("Job Status : ", status['JobRun']['JobRunState'])
    except Exception as e:
        print(e)
        print('Error getting object {} from bucket {}. Make sure they exist '
              'and your bucket is in the same region as this '
              'function.'.format(source_bucket, source_bucket))
    raise e

We have this setup in production environment and running successfully the last 1.5 years.

Thanks,

Yuva

Answer 2

If your source is s3 then you don't even need to run your Glue job every hour to determine if there is any uploads/change to files in source s3 bucket.

You can leverage s3 lambda trigger which will actually trigger your Glue job if there is any upload to s3. Once the lambda triggers then you can start your Glue job. Check out this video for more.

This way you can only run your Glue job only when there is upload instead every hour.

If you still want to run your Glue job every hour then you can use Glue job bookmarking which only process latest data every run.

Answer 3

The psuedocode you have outlined could work as I've run similar jobs in the past.

However, I discovered that Glue jobs used this way are expensive as you will be charged for the 1st 10 mins block of usage even though your job ran for less than a minute (especially in the case where there are no files).

An alternative with better cost (but more complexity as you will use S3 events, SQS, and Lambda in concert) is to do the following:

1. Setup an Events notification in S3 watching for PUT events in the relevant folder that will send a message to SQS (Simple Queue Service).
2. Set the SQS queue's message retention period to 1 hour (or whatever period you are running the Glue job). This way, messages are only in the queue for up to 1 hour.
3. Create a Lambda job that will check the SQS queue for messages (using boto3). Basically you will put the pseudocode you have in Lambda instead of Glue. If there are messages (which means at least 1 file has arrived in that period), trigger the Glue job to process. If not, do thing and exit.

The above method will save you $$.