AWS region in AWS Glue

Question

How can I get the region in which the current Glue job is executing?

---

When the Glue job starts executing, I see the output

Detected region eu-central-1 .

In AWS Lambda, I can use the following lines to fetch the current region:

import os
region = os.environ['AWS_REGION']

However, it seems like the AWS_REGION environment variable is not present in Glue and therefore a KeyError is raised:

KeyError: 'AWS_REGION'

---

The reason why I need the region is I am trying to fetch all databases and tables as described in this question and I do not want to hard code the region when creating the boto client.

Answer 1

One option is to pass the AWS_REGION as a job parameter. For example, if you trigger the job from Lambda:

import os

response = client.start_job_run(
    JobName = 'a_job_name',
    Arguments = {'--AWS_REGION': os.environ['AWS_REGION'] } 
)

Alternatively, if you define your jobs using the AWS::Glue::Job CloudFormation resource:

GlueJob:
  Type: AWS::Glue::Job
  Properties:
    Role: !Ref GlueRole
    DefaultArguments:
      "--AWS_REGION": !Sub "${AWS::Region}"
    Command:
      ScriptLocation: !Sub s3://${GlueScriptBucket}/glue-job.py
      Name: glueetl

Then you can extract the AWS_REGION parameter in your job code using getResolvedOptions :

import sys
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv, ['AWS_REGION'])
print('region', args['AWS_REGION'])

Answer 2

Use os.environ['AWS_DEFAULT_REGION'] instead.

Leaving this here for new visitors.

Answer 3

os.environ['AWS_DEFAULT_REGION'] works for Glue versions 2.0 and 3.0 , but does not exist in previous versions. It gives the region code, for example us-east-1 .

This was confirmed through running a small PySpark script to print out the environment variables, on all the Glue versions, as suggested by the other answer.