从本地 Spark 访问 AWS Glue

Question

Is there any way to run local master Spark SQL queries against AWS Glue?有没有办法针对 AWS Glue 运行本地主 Spark SQL 查询？

Launch this code on my local PC:在我的本地 PC 上启动此代码：

SparkSession.builder()
    .master("local")
    .enableHiveSupport()
    .config("hive.metastore.client.factory.class", "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory")
    .getOrCreate()
    .sql("show databases"); // this query isn't running against AWS Glue

EDIT based on some examples it appears that the hive.metastore.uris configuration key should allow specifying a specific metastore url, however, it's not clear how to get the relevant value for glue根据一些示例进行编辑，似乎hive.metastore.uris配置键应该允许指定特定的 Metastore url，但是，尚不清楚如何获取胶水的相关值

SparkSession.builder()
    .master("local")
    .enableHiveSupport()
    .config("hive.metastore.client.factory.class", "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory")
    .config("hive.metastore.uris", "thrift://???:9083")
    .getOrCreate()
    .sql("show databases"); // this query isn't running against AWS Glue

Answer 1

Amazon provide this client that should solve the problem.亚马逊提供了这个应该可以解决问题的客户端。 (didn't try it yet) （还没试过）

https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore

从本地 Spark 访问 AWS Glue

问题描述

1 个解决方案

解决方案1
3 2019-09-16 20:18:21

从本地 Spark 访问 AWS Glue

问题描述

1 个解决方案

解决方案1 3 2019-09-16 20:18:21

解决方案1
3 2019-09-16 20:18:21