[英]Is there a way to programmatically convert JSON to AVRO Schema?
我需要創建 AVRO 文件,但為此我需要兩件事:
1) JSON
2) Avro 架構
從這兩個要求 - 我有 JSON:
{"web-app": {
"servlet": [
{
"servlet-name": "cofaxCDS",
"servlet-class": "org.cofax.cds.CDSServlet",
"init-param": {
"configGlossary:installationAt": "Philadelphia, PA",
"configGlossary:adminEmail": "ksm@pobox.com",
"configGlossary:poweredBy": "Cofax",
"configGlossary:poweredByIcon": "/images/cofax.gif",
"configGlossary:staticPath": "/content/static",
"templateProcessorClass": "org.cofax.WysiwygTemplate",
"templateLoaderClass": "org.cofax.FilesTemplateLoader",
"templatePath": "templates",
"templateOverridePath": "",
"defaultListTemplate": "listTemplate.htm",
"defaultFileTemplate": "articleTemplate.htm",
"useJSP": false,
"jspListTemplate": "listTemplate.jsp",
"jspFileTemplate": "articleTemplate.jsp",
"cachePackageTagsTrack": 200,
"cachePackageTagsStore": 200,
"cachePackageTagsRefresh": 60,
"cacheTemplatesTrack": 100,
"cacheTemplatesStore": 50,
"cacheTemplatesRefresh": 15,
"cachePagesTrack": 200,
"cachePagesStore": 100,
"cachePagesRefresh": 10,
"cachePagesDirtyRead": 10,
"searchEngineListTemplate": "forSearchEnginesList.htm",
"searchEngineFileTemplate": "forSearchEngines.htm",
"searchEngineRobotsDb": "WEB-INF/robots.db",
"useDataStore": true,
"dataStoreClass": "org.cofax.SqlDataStore",
"redirectionClass": "org.cofax.SqlRedirection",
"dataStoreName": "cofax",
"dataStoreDriver": "com.microsoft.jdbc.sqlserver.SQLServerDriver",
"dataStoreUrl": "jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon",
"dataStoreUser": "sa",
"dataStorePassword": "dataStoreTestQuery",
"dataStoreTestQuery": "SET NOCOUNT ON;select test='test';",
"dataStoreLogFile": "/usr/local/tomcat/logs/datastore.log",
"dataStoreInitConns": 10,
"dataStoreMaxConns": 100,
"dataStoreConnUsageLimit": 100,
"dataStoreLogLevel": "debug",
"maxUrlLength": 500}},
{
"servlet-name": "cofaxEmail",
"servlet-class": "org.cofax.cds.EmailServlet",
"init-param": {
"mailHost": "mail1",
"mailHostOverride": "mail2"}},
{
"servlet-name": "cofaxAdmin",
"servlet-class": "org.cofax.cds.AdminServlet"},
{
"servlet-name": "fileServlet",
"servlet-class": "org.cofax.cds.FileServlet"},
{
"servlet-name": "cofaxTools",
"servlet-class": "org.cofax.cms.CofaxToolsServlet",
"init-param": {
"templatePath": "toolstemplates/",
"log": 1,
"logLocation": "/usr/local/tomcat/logs/CofaxTools.log",
"logMaxSize": "",
"dataLog": 1,
"dataLogLocation": "/usr/local/tomcat/logs/dataLog.log",
"dataLogMaxSize": "",
"removePageCache": "/content/admin/remove?cache=pages&id=",
"removeTemplateCache": "/content/admin/remove?cache=templates&id=",
"fileTransferFolder": "/usr/local/tomcat/webapps/content/fileTransferFolder",
"lookInContext": 1,
"adminGroupID": 4,
"betaServer": true}}],
"servlet-mapping": {
"cofaxCDS": "/",
"cofaxEmail": "/cofaxutil/aemail/*",
"cofaxAdmin": "/admin/*",
"fileServlet": "/static/*",
"cofaxTools": "/tools/*"},
"taglib": {
"taglib-uri": "cofax.tld",
"taglib-location": "/WEB-INF/tlds/cofax.tld"}}}
但是如何基於它創建 AVRO Schema?
尋找編程方式來做到這一點,因為將有很多架構並且不能每次都手動創建 Avro 架構。
我檢查了“avro-tools-1.8.1.jar”,但不能直接從 JSON 創建 Avro Schema。
尋找可以創建 JSON -> Avro 模式的 Jar 或 Python 代碼。 如果數據類型不完美也沒關系(字符串、整數和浮點數足以開始)。
您可以使用 Kite SDK util 從 json 輸入推斷 avro 模式。
示例:
String json = "{\n" +
" \"id\": 1,\n" +
" \"name\": \"A green door\",\n" +
" \"price\": 12.50,\n" +
" \"tags\": [\"home\", \"green\"]\n" +
"}\n"
;
String avroSchema = JsonUtil.inferSchema(JsonUtil.parse(json), "myschema").toString();
System.out.println(avroSchema);
結果:
{
"type":"record",
"name":"myschema",
"fields":[
{
"name":"id",
"type":"int",
"doc":"Type inferred from '1'"
},
{
"name":"name",
"type":"string",
"doc":"Type inferred from '\"A green door\"'"
},
{
"name":"price",
"type":"double",
"doc":"Type inferred from '12.5'"
},
{
"name":"tags",
"type":{
"type":"array",
"items":"string"
},
"doc":"Type inferred from '[\"home\",\"green\"]'"
}
]
}
您可以在 此處找到 maven 依賴項
通過簡單的 avro 模式復制和粘貼,這個很酷。
https://toolslick.com/generation/metadata/avro-schema-from-json
如果您想避免為每種 JSON 格式創建專用的 AVRO 模式,您可以使用rec-avro
包。
它允許您采用任何 Python 數據結構,包括解析的 XML 或 JSON,並將其存儲在 Avro 中,而無需專用架構。
我為python 3測試了它。
您可以將其安裝為 pip3 install rec-avro 或在https://github.com/bmizhen/rec-avro查看代碼和文檔
我在這里給了一個 json 給 avro 示例代碼: https ://stackoverflow.com/a/55444481/6654219
試一試。 http://www.dataedu.ca/avro
它基本上推斷接受 JSON 的 Avro 模式。
你甚至可以給它一個 JSON 數組。 它會做的是生成一個與數組中的所有JSON 文檔兼容的 Avro 模式。
還有其他工具可以驗證結果。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.