频谱扫描错误：第 0 列使用字典编码，但字典页面为空

Question

I just started using AWS Glue V4.0 to generate Glue catalog tables and store the data as parquet files on S3. I use Redshift Spectrum to create an external table which allows me to read this S3 data directly from Redshift.

This was working with Glue V3.0, but since the upgrade to Glue V4.0, I am getting this error (edited to hide the S3 path).

Error: Spectrum Scan Error.
Code: 15007
Context: Parquet file 
Parquet file 'https://s3.us-east-1.amazonaws.com/<<bucket>>/<<path>>/created_date%3D2021-12-22/run-1670477101661-part-block-0-0-r-00000-snappy.parquet': 
metadata is corrupt. Column 0 uses dictionary encoding but the dictionary page is empty. 
(s3://<<bucket>>/<<path>>/created_date=2021-12-22/run-1670477101661-part-block-0-0-r-00000-snappy.parquet)
query: 42738706
location: dory_util.cpp:1445
process: worker_thread [pid=8836]

I can query the data in Athena, but not in Redshift. I can also read and query the parquet file if I read it in a local Spark session.

I tried generating the parquet files using two approaches, each without luck:

1. glue_context.getSink using glueparquet format;
2. glue_context.write_dynamic_frame_from_catalog , using parquet format and setting useGlueParquetWriter to true .

The external schema in Redshift was created like this:

create external schema if not exists my_ext_database from data catalog database 'my_ext_database'
    iam_role 'arn:aws:iam::123456789:role/my-role-name';

I was expecting to be able to query the external schema from Redshift. Why can't Redshift Spectrum read the data?

Answer 1

I've solved it by using pyspark to write the parquet files to S3: basically something like df.write.parquet('s3://<bucket>/<path>/') . Now I can access the data in Redshift using an external schema (via Redshift Spectrum).

This also implies that I have had to create a Glue catalog table to point to the path. And I also need to maintain the partitions in the Glue catalog table. I used boto3 to automate those: but it was a fair bit of work to develop.

So basically my solution is to avoid using getSink or write_dynamic_frame_from_catalog with Glue 4.0 to produce the parquet data, as these methods produce parquet files which, at the time of writing this, cannot be read by Redshift Spectrum.

Despite the workaround, I have a couple of remaining (but non-breaking) drawbacks:

1. Redshift introspection fails in Pycharm. This is fixed in PyCharm 2022.3.1 RC.
2. Clicking on a Glue table to view its schema in the AWS console yields the error, This table no longer exists .
   • One workaround is to open the table in a new browser tab (using Chrome) and then the full schema is displayed.
   • Another workaround is to trick AWS Glue by specifying the glueparquet parameter in the partition storage descriptor (ie {'StorageDescriptor': {'Parameters': {'classification': 'glueparquet'}}} ).