分区依据的 Amazon Redshift Spectrum 不返回结果

Question

Given an S3 bucket partitioned by date in this way:

year
|___month
    |___day
        |___file_*.parquet

I am trying to create a table in amazon redshift Spectrum with this command:

create external table spectrum.visits(
    ip varchar(100),
  user_agent varchar(2000),
  url varchar(10000),
  referer varchar(10000),
  session_id char(32),
  store_id int,
  category_id int,
  page_id int,
  product_id int,
  customer_id int,
  hour int
)
partitioned by (year char(4), month varchar(2), day varchar(2))
stored as parquet
location 's3://visits/visits-parquet/';

Although an error message is not thrown, the results of the queries are always null, ie, do not return results. The bucket is not null. Does someone knows want am I doing wrong?

Answer 1

When an External Table is created in Amazon Redshift Spectrum, it does not scan for existing partitions . Therefore, Redshift is not aware that they exist.

You will need to execute an ALTER TABLE... ADD PARTITION command for each existing partition.

(Amazon Athena has a MSCK REPAIR TABLE option, but Redshift Spectrum does not.)

Answer 2

As I can't comment on people solutions, I needed to add another one.

I would like to point that if your spectrum table comes from Amazon Glue Data Catalog you don't need to manually add partitions to tables, you can have a crawler update partitions on the data catalog and the changes will reflect on spectrum.

Answer 3

One can create external table in Athena & run msck repair on it. Make sure you add "/" at the end of the location. Then create external schema in redshift. This solved my problem of result being showing blank. Alternatively you can run Glue crawler on Athena database, that will generate partitions automatically.