使用 boto3 和 athena 读取 python 中的 alb 日志

Question

我想在我的 django 代码中访问 alb 日志，我正在尝试使用 athena 查询来获取我正在使用 boto3 版本 1.3.1 连接到 athena 客户端的日志，以使用 python 查询到 athena

这是我在 python 版本 3.5 中的代码

import time
import boto3

query = """
            SELECT *
            from alb_logs
            """
DATABASE = 'alb_logs'
output='s3://bucket/'

client = boto3.client('athena')
# Execution
response = client.start_query_execution(
    QueryString=query,
    QueryExecutionContext={
        'Database': DATABASE
    },
    ResultConfiguration={
        'OutputLocation': output,
    }
)
print(response)

但我不能雅典娜它给 UnknownServiceError 这是我得到的错误

UnknownServiceError: Unknown service: 'athena'. Valid service names are: acm, apigateway, application-autoscaling, appstream, autoscaling, batch, budgets, cloudformation, cloudfront, cloudhsm, cloudsearch, cloudsearchdomain, cloudtrail, cloudwatch, codebuild, codecommit, codedeploy, codepipeline, cognito-identity, cognito-idp, cognito-sync, config, datapipeline, devicefarm, directconnect, discovery, dms, ds, dynamodb, dynamodbstreams, ec2, ecr, ecs, efs, elasticache, elasticbeanstalk, elastictranscoder, elb, elbv2, emr, es, events, firehose, gamelift, glacier, health, iam, importexport, inspector, iot, iot-data, kinesis, kinesisanalytics, kms, lambda, lightsail, logs, machinelearning, marketplacecommerceanalytics, meteringmarketplace, opsworks, opsworkscm, pinpoint, polly, rds, redshift, rekognition, route53, route53domains, s3, sdb, servicecatalog, ses, shield, sms, snowball, sns, sqs, ssm, stepfunctions, storagegateway, sts, support, swf, waf, waf-regional, workspaces, xray

这是错误的回溯：

---------------------------------------------------------------------------
UnknownServiceError                       Traceback (most recent call last)
<ipython-input-208-8eaf965f48e5> in <module>
     16 output='s3://sr-prod-alb-logs-mumbai/'
     17 
---> 18 client = boto3.client('athena')
     19 # Execution
     20 response = client.start_query_execution(

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/boto3/__init__.py in client(*args, **kwargs)
     77     See :py:meth:`boto3.session.Session.client`.
     78     """
---> 79     return _get_default_session().client(*args, **kwargs)
     80 
     81 def resource(*args, **kwargs):

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/boto3/session.py in client(self, service_name, region_name, api_version, use_ssl, verify, endpoint_url, aws_access_key_id, aws_secret_access_key, aws_session_token, config)
    248             aws_access_key_id=aws_access_key_id,
    249             aws_secret_access_key=aws_secret_access_key,
--> 250             aws_session_token=aws_session_token, config=config)
    251 
    252     def resource(self, service_name, region_name=None, api_version=None,

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/botocore/session.py in create_client(self, service_name, region_name, api_version, use_ssl, verify, endpoint_url, aws_access_key_id, aws_secret_access_key, aws_session_token, config)
    826             is_secure=use_ssl, endpoint_url=endpoint_url, verify=verify,
    827             credentials=credentials, scoped_config=self.get_scoped_config(),
--> 828             client_config=config, api_version=api_version)
    829         return client
    830 

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/botocore/client.py in create_client(self, service_name, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, api_version, client_config)
     60                       api_version=None,
     61                       client_config=None):
---> 62         service_model = self._load_service_model(service_name, api_version)
     63         cls = self._create_client_class(service_name, service_model)
     64         endpoint_bridge = ClientEndpointBridge(

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/botocore/client.py in _load_service_model(self, service_name, api_version)
     90     def _load_service_model(self, service_name, api_version=None):
     91         json_model = self._loader.load_service_model(service_name, 'service-2',
---> 92                                                      api_version=api_version)
     93         service_model = ServiceModel(json_model, service_name=service_name)
     94         self._register_retries(service_model)

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/botocore/loaders.py in _wrapper(self, *args, **kwargs)
    130         if key in self._cache:
    131             return self._cache[key]
--> 132         data = func(self, *args, **kwargs)
    133         self._cache[key] = data
    134         return data

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/botocore/loaders.py in load_service_model(self, service_name, type_name, api_version)
    376             raise UnknownServiceError(
    377                 service_name=service_name,
--> 378                 known_service_names=', '.join(sorted(known_services)))
    379         if api_version is None:
    380             api_version = self.determine_latest_version(

后续问题：如果我使用 gzip 文件读取 alb-logs 并使用文档中提供的正则表达式将其制成所需的格式 ( https://docs.aws.amazon.com/athena/latest/ug/application-load -balancer-logs.html ），如果亚马逊将来在 alb 日志中添加更多列或者我的正则表达式会中断，此代码是否仍然有效？ 有没有其他更好的方法来读取 python 中的 alb-logs？

Answer 1

我会说你的boto3已经过时了。 当前版本为1.26.16 （2022 年 11 月）。

您使用的版本是 2016 年 7 月的版本，Athena 是 2016 年 11 月才发布的。