[英]AWS Glue job fails to write to Redshift
I have an AWS Glue job that should write the results from a dynamic frame to a Redshift database.我有一个 AWS Glue 作业,应该将动态框架中的结果写入 Redshift 数据库。 For that I have set up a Glue Connection to Redshift and tested it, and it works fine.
为此,我设置了一个到 Redshift 的胶水连接并对其进行了测试,它工作正常。 However, when the job tries to write to Redshift it timeouts with
但是,当作业尝试写入 Redshift 时,它会超时
AnalysisException: 'java.lang.RuntimeException: com.amazonaws.SdkClientException:
Unable to execute HTTP request:
Connect to glue.eu-central-1.amazonaws.com:443 [glue.eu-central-1.amazonaws.com/52.57.157.164,
glue.eu-central-1.amazonaws.com/35.156.122.139,
glue.eu-central-1.amazonaws.com/3.122.136.8] failed: connect timed out;'
The code that is used for writing looks like this:用于编写的代码如下所示:
my_conn_options = {
"dbtable": "tbl1",
"database": "db1",
"aws_iam_role": "arn:aws:iam::111111111111:role/glue-role"
}
glueContext.write_dynamic_frame.from_jdbc_conf(
frame = dynamic_frame,
catalog_connection = "glue-to-redshift-connection",
connection_options = my_conn_options,
redshift_tmp_dir = args["TempDir"])
Any Idea what could be causing that?任何想法可能是什么原因造成的?
Once you use a Glue Connection within the job, the job is executed within your VPC and external connections no longer work, eg to glue.eu-central-1.amazonaws.com.在作业中使用 Glue 连接后,作业将在您的 VPC 中执行,并且外部连接不再起作用,例如,glue.eu-central-1.amazonaws.com。
In order to make internal traffic possible, you need to create a VPC endpoint for Glue.为了使内部流量成为可能,您需要为 Glue 创建一个VPC 终端节点。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.