AWS EMR Spark Glue PySpark -

Question

我正在嘗試將 AWS Glue 表讀入 pyspark。 我得到一個空指針異常：

spark.sql("show tables").show()
+----------------+-----------------+-----------+
|        database|        tableName|isTemporary|
+----------------+-----------------+-----------+
|test_datalake_db|events2_2017_test|      false|
|test_datalake_db|      events2_old|      false|
+----------------+-----------------+-----------+

接下來，我嘗試從表中選擇一些內容：

df = spark.sql("select * from events2_2017_test")

然而，事情變得一團糟：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/spark/python/pyspark/sql/session.py", line 603, in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
  File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/usr/lib/spark/python/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o51.sql.
: java.lang.NullPointerException: Name is null
    at java.lang.Enum.valueOf(Enum.java:236)

也失敗了，比如：

myDf = spark.table("test_datalake_db.events2_2017_test")

這是表架構：

Answer 1

答案是您需要在創建表時將 TableType 定義為 EXTERNAL_TABLE：

def create_table():
    response = client.create_table(
        DatabaseName='xxxx-datalake',
        TableInput={
            'Name': 'events2_2017_sample',
            'Description': 'testing the events2 old data in s3',
            'Owner': 'drew',
            'Parameters': {
                "classification": "csv"
            },
            'TableType': 'EXTERNAL_TABLE',
            'StorageDescriptor': {
                'Columns': [
                    {'Name': 'sys_vortex_id', 'Type': 'string'},
                    {'Name': 'sys_app_id', 'Type': 'string'},
                    {'Name': 'sys_pq_id', 'Type': 'string'},
                    {'Name': 'sys_ip_address', 'Type': 'string'},
                    {'Name': 'sys_submitted_at', 'Type': 'string'},
                    {'Name': 'sys_received_at', 'Type': 'string'},
                    {'Name': 'device_id_type', 'Type': 'string'},
                    {'Name': 'device_id', 'Type': 'string'},
                    {'Name': 'timezone', 'Type': 'string'},
                    {'Name': 'online', 'Type': 'string'},
                    {'Name': 'app_version', 'Type': 'string'},
                    {'Name': 'device_days', 'Type': 'string'},
                    {'Name': 'device_sessions', 'Type': 'string'},
                    {'Name': 'event_id', 'Type': 'string'}
                ],
                'InputFormat': 'org.apache.hadoop.mapred.TextInputFormat',
                'OutputFormat': 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat',
                'SerdeInfo': {
                    'Name': 'noidea',
                    'SerializationLibrary': 'org.apache.hadoop.hive.serde2.OpenCSVSerde',
                    'Parameters': {
                        'separatorChar': '|'
                    }
                },
                'Location': 's3://xxxx-datalake-us-east-1/prd/101/27a89ba8-774a-4bc3-68ae/laboratory/events2_2017_test/',
                'Compressed': True,
                'StoredAsSubDirectories': False,
            }
        }
    )