在 AWS Glue Python 腳本中從 Secrets Manager 檢索 RDS 憑證

Question

我有一個 Glue 腳本，它試圖讀取我存儲在 Secrets manager 中的 RDS 憑據。 但是腳本繼續運行並且永遠不會完成。 此外，運行此 Glue 腳本的 IAM 角色包含SecretsManagerReadWrite策略（AWS 托管）

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.dynamicframe import DynamicFrameCollection
from awsglue.dynamicframe import DynamicFrame
import boto3
import botocore
from botocore.errorfactory import ClientError
# import org.apache.spark.sql.functions.concat_ws
from pyspark.sql.types import *
from pyspark.sql.functions import udf
from datetime import date
today = date.today()
current_day = today.strftime("%Y%m%d")

def str_to_arr(my_list):
    str = ""
    for item in my_list:
        if item:
            str += item
    str = str.split(" ")
    return '{"' + ' '.join([elem for elem in str])  + '"}'

str_to_arr_udf = udf(str_to_arr,StringType())

def AddPartitionKeys(glueContext, dfc) -> DynamicFrameCollection:
    df = dfc.select(list(dfc.keys())[0]).toDF()
    df = glueContext.add_ingestion_time_columns(df, "day")
    glue_df = DynamicFrame.fromDF(df, glueContext, "transform_date")
    return(DynamicFrameCollection({"CustomTransform0": glue_df}, glueContext))

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME', 'days', 's3_bucket', 'rds_endpoint', 'region_name', 'secret_name'])

region_name = args['region_name']
session = boto3.session.Session()
client = session.client("secretsmanager", region_name=region_name)
get_secret_value_response = client.get_secret_value(SecretId=args['secret_name'])
secret = get_secret_value_response['SecretString']
secret = json.loads(secret)
db_username = secret.get('username')
db_password = secret.get('password')
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
print("Below are the creds")
# print("DB USERNAME IS " , db_username)
# print("DB PWD IS " , db_password)
job = Job(glueContext)

job.init(args['JOB_NAME'], args)

job.commit()

我在這里想念什么？

我對照這個博客檢查了我的工作，但我無法成功完成這個腳本。

Answer 1

在 Mark 的建議下，我發現我必須為 Secrets Manager 創建一個 VPC 接口端點。 AWS 在此處概述了這些步驟，只需確保終端節點中的策略具有提及我想從 Secrets Manager 訪問的資源的訪問權限/ARN。