![](/img/trans.png)
[英]Write avro files with LogicalType 'timestamp-millis' on date/timestamps in Java Beam pipeline
[英]How to serialize timestamp-millis logicalType to avro file using java
我正在創建要創建具有時間戳列的avro文件的場景,該列看起來像2016-11-16 06:43:19.77
我已使用avro-1.8.2.jar編寫avro文件。 這對於原始的記錄類型非常適合,但對於像timestamp-millis這樣的邏輯類型則非常適用。
架構:
{
"type": "record",
"name": "MyRecord",
"namespace": "org.demo",
"fields": [
{
"name": "timestamp_with_logical_type",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
}
},
{
"name": "timestamp_no_logical_type",
"type": "long"
}
]
}
Java代碼:
//Instantiating the GenericRecord class.
GenericRecord record = new Record(schema);
long millis = Instant.now().toEpochMilli();
//Insert data according to schema
record.put("timestamp_with_logical_type", new Timestamp(millis));
record.put("timestamp_no_logical_type", millis);
DataFileWriter<GenericRecord> dataFileWriter = null;
DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<GenericRecord>(schema);
dataFileWriter = new DataFileWriter<GenericRecord>(datumWriter);
dataFileWriter.setCodec(CodecFactory.snappyCodec());
dataFileWriter.setFlushOnEveryBlock(true);
dataFileWriter.setSyncInterval(32);
dataFileWriter.create(SCHEMA,new File("E:\\MyFiles\\Avro_Data\\demo.avro"));
dataFileWriter.append(record);
dataFileWriter.close();
上面的代碼給出以下異常:
Exception in thread "main" org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to java.lang.CharSequence
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
at AvroFileReaderWriter.writeToAvro(AvroFileReaderWriter.java:264)
at AvroFileReaderWriter.main(AvroFileReaderWriter.java:74)
Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to java.lang.CharSequence
at org.apache.avro.generic.GenericDatumWriter.writeString(GenericDatumWriter.java:267)
at org.apache.avro.generic.GenericDatumWriter.writeString(GenericDatumWriter.java:262)
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:128)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:166)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:62)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
... 2 more
我認為您需要使用Instant而不是Timestamp,因此在Java代碼中:
Java代碼:
//Instantiating the GenericRecord class.
GenericRecord record = new Record(schema);
Instant millis = Instant.now();
//Insert data according to schema
record.put("timestamp_with_logical_type", millis));
record.put("timestamp_no_logical_type", millis.toEpochMilli());
DataFileWriter<GenericRecord> dataFileWriter = null;
DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<GenericRecord>(schema);
dataFileWriter = new DataFileWriter<GenericRecord>(datumWriter);
dataFileWriter.setCodec(CodecFactory.snappyCodec());
dataFileWriter.setFlushOnEveryBlock(true);
dataFileWriter.setSyncInterval(32);
dataFileWriter.create(SCHEMA,new File("E:\\MyFiles\\Avro_Data\\demo.avro"));
dataFileWriter.append(record);
dataFileWriter.close();
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.