ATHENA CREATE TABLE AS 镶木地板格式问题

Question

I'm creating a table in Athena and specifying the format as PARQUET however the file extension is not being recognized in S3. The type is displayed as "-" which means that the file extension is not recognized despite that I can read the files (written from Athena) successfully in a Glue job using:

df = spark.read.parquet()

Here is my statement:

CREATE EXTERNAL TABLE IF NOT EXISTS test (
    numeric_field INT
    ,numeric_field2 INT)

STORED AS PARQUET
LOCATION 's3://xxxxxxxxx/TEST TABLE/'
TBLPROPERTIES ('classification'='PARQUET');   
    
INSERT INTO test
VALUES (10,10),(20,20);

I'm specifying the format as PARQUET but when I check in the S3 bucket the file type is displayed as "-". Also when I check the glue catalog, that table type is set as 'unknown'

S3 STORAGE PRINT SCREEN

I expected that the type is recognized as "parquet" in the S3 bucket

Answer 1

After contacting the AWS support, it was confirmed that with CTAS queries Athena does not create file extensions for parquet files. "Further to confirm this, I do see the Knowledge Center article [1] where CTAS generates the Parquet files without extension ( Under section 'Convert the data format and set the approximate file size' Point 5)."

However the files written from Athena, even without the extension are readable.

Reference: [1] https://aws.amazon.com/premiumsupport/knowledge-center/set-file-number-size-ctas-athena/

Workaround: I created a function to change the file extension. Basically iterating over the files in the S3 bucket and then writing the contents back to the same location with parquet file extension