[英]Generate Avro Schema for Java POJO with Generic Types
我正在嘗試使用以下方法在運行時獲取 Avro Schema:
private Schema getSchema(Class clazz) {
Schema s = ReflectData.get().getSchema(clazz);
AvroSchema avroSchema = new AvroSchema(s);
return avroSchema.getAvroSchema();
}
但是由於我的 POJO 類包含如下泛型:
public abstract class Data<T> implements Serializable {
private static final long serialVersionUID = 1L;
private String dataType;
private T id;
public Data() {
}
public Data(String dataType) {
this.dataType = dataType;
}
public Data(String dataType, T id) {
this.dataType = dataType;
this.id = id;
}
}
我收到以下異常:
Exception in thread "main" org.apache.avro.AvroRuntimeException: avro.shaded.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.avro.AvroTypeException: Unknown type: T
at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:227)
我知道 Avro 不支持泛型類型。 有沒有辦法可以在運行時生成模式期間從我的類中省略某些類字段?
private <T> String writePojoToParquet(List<T> pojos, String fileKey){
String fileName = fileKey + ".parquet";
Path path = new Path(fileName.replace("/", "_"));
//No matter what delete file always.
String strPath = path.toString();
FileUtils.delete(strPath);
FileUtils.delete(strPath + ".crc");
logger.debug("Writing data to parquet file {}", strPath);
Configuration conf = new Configuration();
try (ParquetWriter<T> writer =
AvroParquetWriter.<T>builder(path)
.withSchema(ReflectData.AllowNull.get().getSchema(pojos.get(0).getClass()))
.withDataModel(ReflectData.get())
.withConf(conf)
.withCompressionCodec(CompressionCodecName.SNAPPY)
.withWriteMode(ParquetFileWriter.Mode.OVERWRITE)
.enableValidation()
.enableDictionaryEncoding()
.build()) {
for (T p : pojos) {
writer.write(p);
}
return strPath;
} catch (IOException e) {
logger.error("Error while writing data to parquet file {}.", strPath, e);
}
return null;
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.