堆栈内存溢出
  繁体   English   中英

将 JSON 文件转换为 AVRO

[英]Convert JSON file to AVRO

 原文  2020-11-05 03:54:37  8   2    python/ json/ avro

问题描述

我是 AVRO 和 Python 的新手。 我有一个用例，我想将 JSON 文件转换为 Avro 文件。 我以 .avsc 格式存储了我的架构，以 .json 格式存储了 JSON 数据。 现在我想将 JSON 文件和 .avsc 文件放在一起，并将 JSON 文件序列化为 Avro。 下面是我的代码，我收到一个错误“avro.io.AvroTypeException: The datum file.json is not an example of the schema”。 我不确定我做错了什么。 在我的 writer.append 语句中，我希望脚本从 JSON 文件中获取数据并将其附加到 .avro 文件中，但采用序列化格式。 我不确定如何处理这个问题，任何帮助将不胜感激。

这是我的代码：

import avro.schema
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter
from avro import schema, datafile, io
import json
from avro import schema, datafile, io


def json_to_avro():
        fo = open("laird.txt", "r")
        data = fo.readlines()
        final_header = []
        final_rec = []
        for header in data[0:1]:
            header = header.strip("\n")
            header = header.split(",")
            final_header = header
        for rec in data[1:]:
            rec = rec.strip("\n")
            rec = rec.split(" ")
            rec = ' '.join(rec).split()
            final_rec = rec
        final_dict = dict(zip(final_header,final_rec))
        # print(final_dict)
        json_dumps = json.dumps(final_dict, ensure_ascii=False)
        # print(json_dumps)
        schema = avro.schema.parse(open("laird.avsc", "rb").read())
        # print(schema)

        writer = DataFileWriter(open("laird.avro", "wb"), DatumWriter(), schema)

        with open("laird.json") as fp:
            contents = json.load(fp)
            print(contents)

        writer.append(contents)

        writer.close()

json_to_avro()

laird.avsc 文件中的数据：

{
  "name": "MyClass",
  "type": "record",
  "namespace": "com.acme.avro",
  "fields": [
    {
      "name": "event_type",
      "type": "string"
    },
    {
      "name": "event_data",
      "type": {
        "name": "event_data",
        "type": "record",
        "fields": [
          {
            "name": "device_id",
            "type": "string"
          },
          {
            "name": "user_id",
            "type": "string"
          },
          {
            "name": "payload",
            "type": {
              "type": "array",
              "items": {
                "name": "payload_record",
                "type": "record",
                "fields": [
                  {
                    "name": "name",
                    "type": "string"
                  },
                  {
                    "name": "sensor_id",
                    "type": "string"
                  },
                  {
                    "name": "type",
                    "type": "string"
                  },
                  {
                    "name": "unit",
                    "type": "string"
                  },
                  {
                    "name": "value",
                    "type": "float"
                  },
                  {
                    "name": "channel",
                    "type": "int"
                  },
                  {
                    "name": "timestamp",
                    "type": "long"
                  }
                ]
              }
            }
          },
          {
            "name": "client_id",
            "type": "string"
          },
          {
            "name": "hardware_id",
            "type": "string"
          },
          {
            "name": "timestamp",
            "type": "long"
          },
          {
            "name": "application_id",
            "type": "string"
          },
          {
            "name": "device_type_id",
            "type": "string"
          }
        ]
      }
    },
    {
      "name": "company",
      "type": {
        "name": "company",
        "type": "record",
        "fields": [
          {
            "name": "id",
            "type": "int"
          },
          {
            "name": "address",
            "type": "string"
          },
          {
            "name": "city",
            "type": "string"
          },
          {
            "name": "country",
            "type": "string"
          },
          {
            "name": "created_at",
            "type": "int",
            "logicalType": "date"
          },
          {
            "name": "industry",
            "type": "string"
          },
          {
            "name": "latitude",
            "type": "float"
          },
          {
            "name": "longitude",
            "type": "float"
          },
          {
            "name": "name",
            "type": "string"
          },
          {
            "name": "state",
            "type": "string"
          },
          {
            "name": "status",
            "type": "int"
          },
          {
            "name": "timezone",
            "type": "string"
          },
          {
            "name": "updated_at",
            "type": "int",
            "logicalType": "date"
          },
          {
            "name": "user_id",
            "type": "string"
          },
          {
            "name": "zip",
            "type": "string"
          }
        ]
      }
    },
    {
      "name": "location",
      "type": {
        "name": "location",
        "type": "record",
        "fields": [
          {
            "name": "id",
            "type": "int"
          },
          {
            "name": "address",
            "type": "string"
          },
          {
            "name": "city",
            "type": "string"
          },
          {
            "name": "country",
            "type": "string"
          },
          {
            "name": "created_at",
            "type": "int",
            "logicalType": "date"
          },
          {
            "name": "industry",
            "type": "string"
          },
          {
            "name": "latitude",
            "type": "float"
          },
          {
            "name": "longitude",
            "type": "float"
          },
          {
            "name": "name",
            "type": "string"
          },
          {
            "name": "state",
            "type": "string"
          },
          {
            "name": "status",
            "type": "int"
          },
          {
            "name": "timezone",
            "type": "string"
          },
          {
            "name": "updated_at",
            "type": "int",
            "logicalType": "date"
          },
          {
            "name": "user_id",
            "type": "string"
          },
          {
            "name": "zip",
            "type": "string"
          },
          {
            "name": "company_id",
            "type": "int"
          }
        ]
      }
    },
    {
      "name": "device_type",
      "type": {
        "name": "device_type",
        "type": "record",
        "fields": [
          {
            "name": "id",
            "type": "string"
          },
          {
            "name": "application_id",
            "type": "string"
          },
          {
            "name": "category",
            "type": "string"
          },
          {
            "name": "codec",
            "type": "string"
          },
          {
            "name": "data_type",
            "type": "string"
          },
          {
            "name": "description",
            "type": "string"
          },
          {
            "name": "manufacturer",
            "type": "string"
          },
          {
            "name": "model",
            "type": "string"
          },
          {
            "name": "name",
            "type": "string"
          },
          {
            "name": "parent_constraint",
            "type": "string"
          },
          {
            "name": "proxy_handler",
            "type": "string"
          },
          {
            "name": "subcategory",
            "type": "string"
          },
          {
            "name": "transport_protocol",
            "type": "string"
          },
          {
            "name": "version",
            "type": "string"
          },
          {
            "name": "created_at",
            "type": "int",
            "logicalType": "date"
          },
          {
            "name": "updated_at",
            "type": "int",
            "logicalType": "date"
          }
        ]
      }
    },
    {
      "name": "device",
      "type": {
        "name": "device",
        "type": "record",
        "fields": [
          {
            "name": "id",
            "type": "int"
          },
          {
            "name": "thing_name",
            "type": "string"
          },
          {
            "name": "created_at",
            "type": "int",
            "logicalType": "date"
          },
          {
            "name": "updated_at",
            "type": "int",
            "logicalType": "date"
          },
          {
            "name": "status",
            "type": "int"
          }
        ]
      }
    }
  ]
}

laird.json 中的数据

{
  "event_type": "uplink",
  "event_data": {
    "device_id": "fec5d310-88bf-11ea-aadf-2bd85a1038fa",
    "user_id": "ba5e224b-fcc2-4e1b-b2b3-97ef57b9ba60",
    "payload": [
      {
        "name": "Temperature",
        "sensor_id": "fef87bd0-88bf-11ea-ae96-6fae9a4d5562",
        "type": "temp",
        "unit": "c",
        "value": 21.9,
        "channel": 3,
        "timestamp": 1603300033446
      },
      {
        "name": "Humidity",
        "sensor_id": "fef399d0-88bf-11ea-a424-59b141c8d9bf",
        "type": "rel_hum",
        "unit": "p",
        "value": 74,
        "channel": 4,
        "timestamp": 1603300033446
      },
      {
        "name": "Battery",
        "sensor_id": "feef2d00-88bf-11ea-aadf-2bd85a1038fa",
        "type": "batt",
        "unit": "p",
        "value": 100,
        "channel": 5,
        "timestamp": 1603300033446
      },
      {
        "name": "RSSI",
        "sensor_id": "fef658f0-88bf-11ea-ae96-6fae9a4d5562",
        "type": "rssi",
        "unit": "dbm",
        "value": -42,
        "channel": 100,
        "timestamp": 1603300033446
      },
      {
        "name": "SNR",
        "sensor_id": "",
        "type": "snr",
        "unit": "db",
        "value": 10.2,
        "channel": 101,
        "timestamp": 1603300033446
      }
    ],
    "client_id": "1c46a2e0-88b9-11ea-90ae-77c0364d6e36",
    "hardware_id": "0025ca0a00008612",
    "timestamp": 1603300033446,
    "application_id": "shipcomwireless",
    "device_type_id": "16a4d0c0-3edf-11e9-bf5e-a180edbfa9bb"
  },
  "company": {
    "id": 6854,
    "address": "10500 University Center Dr",
    "city": "Tampa",
    "country": "United States",
    "created_at": "2020-04-27T19:44:46Z",
    "industry": "[\"Health Care\"]",
    "latitude": 28.045853,
    "longitude": -82.421,
    "name": "Dermpath Diagnostics Bay Area",
    "state": "FL",
    "status": 0,
    "timezone": "America/New_York",
    "updated_at": "2020-04-27T19:44:46Z",
    "user_id": "ba5e224b-fcc2-4e1b-b2b3-97ef57b9ba60",
    "zip": "33612"
  },
  "location": {
    "id": 8138,
    "address": "10500 University Center Dr",
    "city": "Tampa",
    "country": "United States",
    "created_at": "2020-04-27T19:44:46Z",
    "industry": "[\"Health Care\"]",
    "latitude": 28.045853,
    "longitude": -82.421,
    "name": "Dermpath Diagnostics Bay Area",
    "state": "FL",
    "status": 0,
    "timezone": "America/New_York",
    "updated_at": "2020-10-19T18:22:19Z",
    "user_id": "ba5e224b-fcc2-4e1b-b2b3-97ef57b9ba60",
    "zip": "33612",
    "company_id": 6854
  },
  "device_type": {
    "id": "16a4d0c0-3edf-11e9-bf5e-a180edbfa9bb",
    "application_id": "",
    "category": "module",
    "codec": "lorawan.laird.rs1xx",
    "data_type": "",
    "description": "Temp Sensor",
    "manufacturer": "Laird",
    "model": "RS1xx",
    "name": "Laird Temp & Humidity RS1xx Sensor - IoT in a Box",
    "parent_constraint": "NOT_ALLOWED",
    "proxy_handler": "PrometheusClient",
    "subcategory": "lora",
    "transport_protocol": "lorawan",
    "version": "",
    "created_at": "2019-03-05T00:39:09Z",
    "updated_at": "2019-03-05T00:39:09Z"
  },
  "device": {
    "id": 217200,
    "thing_name": "Slide/Block Warehouse STE250",
    "created_at": "2020-04-27T19:47:58Z",
    "updated_at": "2020-04-28T12:58:08Z",
    "status": 0
  }
}

2 个解决方案

解决方案1
 1  2020-11-05 04:27:03

您可以尝试使用 fastavro 和 rec_avro 模块，这是一些示例

from fastavro import writer, reader, schema
from rec_avro import to_rec_avro_destructive, from_rec_avro_destructive, rec_avro_schema

def json_objects():
    return [{'a': 'a'}, {'b':'b'}]

# For efficiency, to_rec_avro_destructive() destroys rec, and reuses it's
# data structures to construct avro_objects 
avroObjects = (to_rec_avro_destructive(rec) for rec in json_objects())

# store records in avro
with open('json_in_avro.avro', 'wb') as x:
    writer(x, schema.parse_schema(rec_avro_schema()), avroObjects)

#load records from avro
with open('json_in_avro.avro', 'rb') as f_in:
    # For efficiency, from_rec_avro_destructive(rec) destroys rec, and 
    # reuses it's data structures to construct it's output
    loadedJson = [from_rec_avro_destructive(rec) for rec in reader(f_in)]

assert loadedJson == json_objects()

解决方案2
 0 已采纳  2020-11-05 15:26:07

更新：

有两个问题。 第一个是在模式中。 您当前拥有以下内容的任何地方：

      {
        "name": "created_at",
        "type": "int",
        "logicalType": "date"
      },

应该是这样的：

      {
        "name": "created_at",
        "type": {"type": "int", "logicalType": "date"}
      },

修复该问题后，另一个问题出在您的数据中。 当你读入 JSON 文件时， created_at和updated_at字段是字符串，但它们需要是datetime.date对象。 您可以使用以下内容为每个人执行此操作：

from datetime import date
contents["company"]["created_at"] = date.fromisoformat(contents["company"]["created_at"].split("T")[0])

原来的：

writer.append函数不接受 JSON 文件的名称。 相反，它接受 JSON 对象。 所以而不是：

writer.append("laird.json")

你会想要这样的东西：

with open("laird.json") as fp:
    contents = json.load(fp)

writer.append(contents)

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 fastavro - 将 json 文件转换为 avro 文件 使用 Python 将最初是 json 的 avro 文件转换回 json 将 XML 转换为 JSON 或 Avro（在 python 中） 如何在 Python 中将 JSON 字符串转换为 Avro？ 如何在没有架构的情况下将 json 转换为 avro？ 如何在 python 中将 Avro 文件转换为 CSV 文件？ 如何从 avro 文件转换为 json 文件，该文件最初作为原始 json 通过 Postman 到 Z3A580F14C3022038F63F51 发送 使用 python 消费者将 avro 序列化消息转换为 json 如何使用 Python 将 JSON 数据转换为 Avro 格式 如何使用 PySpark 将 csv 文件转换为 avro 文件？
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM