使用 python 在文件中写入 avro 记录

Question

I have the avro file with name test.avsc and with schema as:

{
    "namespace": "test",
    "type": "record",
    "name": "test.schema",
    "fields": [
      { "name": "device", "type": ["null", "string"] , "default" : null }
    ]
}

My avro class is:

import avro.schema
from avro.datafile import DataFileWriter
from avro.io import DatumWriter

class AvroHelper(object):

    avro_writer = None
    codec = 'deflate'

    def __init__(self, schemaf, avrofile):
        schema = avro.schema.parse(schemaf)
        self.avro_writer = DataFileWriter(avrofile, DatumWriter(), schema, codec=self.codec)
        self.num_rows_written_to_avro = 0

    def append(self, row):
        self.avro_writer.append(row)

I am trying to write data as:

file = open('test1.avro', 'wb')
avro_writer = AvroHelper('test.avsc', file)

rec= { 'device': "1" }
avro_writer.append(rec)

I am getting exception as, while trying to write avro record in a file:

File "search_console_api_client.py", line 31, in __init__
    schema = avro.schema.parse(schema_str)
  File "/usr/local/lib/python3.7/site-packages/avro/schema.py", line 1238, in parse
    % (json_string, exn))
avro.schema.SchemaParseException: Error parsing schema from JSON: 'test.avsc'. Error message: JSONDecodeError('Expecting value: line 1 column 1 (char 0)').

How can I make it work?

Answer 1

You are doing

avro_writer = AvroHelper('test.avsc', file)

Which means your __init__ is having this happen:

schema = avro.schema.parse('test.avsc')

However, the parse() function is supposed to take in the JSON string of the schema, not the name of the file. Instead, you probably want to do something like this:

avro_writer = AvroHelper(open('test.avsc').read(), file)