[英]ASW Athena boto3 iceberg table - insert
我将 AWS 与 Athena iceberg 一起使用。 我正在尝试使用 boto3 向 Athena iceberg 表插入新记录,但出现内部错误GENERIC_INTERNAL_ERROR:无法在没有事务冲突的情况下提交。 如果数据清单文件生成于 *- 我猜 boto3 会像常规 Athena 表一样尝试将文件添加到 s3。 有任何想法吗:
...
import boto3
import pandas as pd
def athena_query_to_dataframe(db, s3Bucket, query):
client = boto3.client('athena')
listOfStatus = ['SUCCEEDED', 'FAILED', 'CANCELLED']
listOfInitialStatus = ['RUNNING', 'QUEUED']
print('Starting Query Execution:')
tempS3Path = 's3://{}'.format(s3Bucket)
response = client.start_query_execution(
QueryString = query,
QueryExecutionContext = {
'Database': db
},
ResultConfiguration = {
'OutputLocation': tempS3Path,
}
)
queryExecutionId = response['QueryExecutionId']
print(client.get_query_execution(QueryExecutionId = queryExecutionId))
status = client.get_query_execution(QueryExecutionId = queryExecutionId)['QueryExecution']['Status']['State']
while status in listOfInitialStatus:
status = client.get_query_execution(QueryExecutionId = queryExecutionId)['QueryExecution']['Status']['State']
if status in listOfStatus:
if status == 'SUCCEEDED':
print('Query Succeeded!')
paginator = client.get_paginator('get_query_results')
query_results = paginator.paginate(
QueryExecutionId = queryExecutionId,
PaginationConfig = {'PageSize': 1000}
)
elif status == 'FAILED':
print('Query Failed!')
elif status == 'CANCELLED':
print('Query Cancelled!')
break
print(client.get_query_execution(QueryExecutionId = queryExecutionId))
results = []
rows = []
print('Processing Response')
in_cmd = """ insert into my_iceberg_table values ('aaaaa','bbb');"""
athena_query_to_dataframe('my'db,'my-bck/athena/tables/my_iceberg_table/',in_cmd)
...
这是权限问题...因为这是一个 Iceberg 表,我还需要在表和模式之间粘合目录权限。 所以我添加了这个 IAM 权限:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:GetDatabases",
"glue:CreateDatabase",
"glue:UpdateTable",
"glue:GetTable"
],
"Resource": [
"arn:aws:glue:*:<account>:table/<schema>/<table>",
"arn:aws:glue:*:<account>:schema/AwsDataCatalog",
"arn:aws:glue:*:<account>:database/<schema>",
"arn:aws:glue:*:<account>:catalog"
]
}
]
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.