如何在 AWS Glue/Athena 上使用 AVRO 格式

Question

I've a few topics in Kafka that are writing AVRO files into S3 buckets and I would like to perform some queries on bucket using AWS Athena.

I'm trying to create a table but AWS Glue crawler runs and doesn't add my table (it works if I change file type to JSON). I've tried to create a table from Athena console but it doesn't show support to AVRO file.

Any idea on how to make it work?

Answer 1

I suggest doing it manually and not via Glue. Glue only works for the most basic situations, and this falls outside that, unfortunately.

You can find the documentation on how to create an Avro table here: https://docs.aws.amazon.com/athena/latest/ug/avro.html

The caveat for Avro tables is that you need to specify both the table columns and the Avro schema. This may look weird and redundant, but it's how Athena/Presto works. It needs a schema to know how to interpret the files, and then it needs to know which of the properties in the files you want to expose as columns (and their types, which may or may not match the Avro types).

CREATE EXTERNAL TABLE avro_table (
   foo STRING,
   bar INT
)
ROW FORMAT
SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
WITH SERDEPROPERTIES ('avro.schema.literal' = '
{
  "type": "record",
  "name": "example",
  "namespace": "default",
  "fields": [
    {
      "name": "foo",
      "type": ["null", "string"],
      "default": null
    },
    {
      "name": "bar",
      "type": ["null", "int"],
      "default": null
    }
  ]
}
')
STORED AS AVRO
LOCATION 's3://some-bucket/data/';

Notice how the Avro schema appears as a JSON document inside of a serde property value (single quoted) – the formatting is optional, but makes this example easier to read.

Answer 2

Doing it manually seems to be the way to make it work. Here is some code to generate the Athena schema directly from a literal avro schema. It works with avro-python3 on python3.7 . It is taken from here: https://github.com/dataqube-GmbH/avro2athena (I am the owner of the repo)

from avro.schema import Parse, RecordSchema, PrimitiveSchema, ArraySchema, MapSchema, EnumSchema, UnionSchema, FixedSchema


def create_athena_schema_from_avro(avro_schema_literal: str) -> str:
    avro_schema: RecordSchema = Parse(avro_schema_literal)

    column_schemas = []
    for field in avro_schema.fields:
        column_name = field.name.lower()
        column_type = create_athena_column_schema(field.type)
        column_schemas.append(f"`{column_name}` {column_type}")

    return ', '.join(column_schemas)


def create_athena_column_schema(avro_schema) -> str:
    if type(avro_schema) == PrimitiveSchema:
        return rename_type_names(avro_schema.type)

    elif type(avro_schema) == ArraySchema:
        items_type = create_athena_column_schema(avro_schema.items)
        return f'array<{items_type}>'

    elif type(avro_schema) == MapSchema:
        values_type = avro_schema.values.type
        return f'map<string,{values_type}>'

    elif type(avro_schema) == RecordSchema:
        field_schemas = []
        for field in avro_schema.fields:
            field_name = field.name.lower()
            field_type = create_athena_column_schema(field.type)
            field_schemas.append(f'{field_name}:{field_type}')

        field_schema_concatenated = ','.join(field_schemas)
        return f'struct<{field_schema_concatenated}>'

    elif type(avro_schema) == UnionSchema:
        # pick the first schema which is not null
        union_schemas_not_null = [s for s in avro_schema.schemas if s.type != 'null']
        if len(union_schemas_not_null) > 0:
            return create_athena_column_schema(union_schemas_not_null[0])
        else:
            raise Exception('union schemas contains only null schema')

    elif type(avro_schema) in [EnumSchema, FixedSchema]:
        return 'string'

    else:
        raise Exception(f'unknown avro schema type {avro_schema.type}')


def rename_type_names(typ: str) -> str:
    if typ in ['long']:
        return 'bigint'
    else:
        return typ