I would like to know if there is a way to drop a record/message from a stream Topology?
I have a setup like the following:
builder.stream("my-source-topic")
.map(CustomMapper)
.mapValues(CustomValueMapper)
.filterNot(CustomFilter)
.transformValues(CustomValueTransformer)
.toStream()
Each CustomMapper/CustomFilter etc overrides their respective apply/transform methods they could look like the following, as noted the error might be unrecoverable and this is an ok solution these messages will be handled manually and a respective log is written. Assuming the unrecoverable error happens during the first map how do i now prevent the later Stages from even processing the record, i would like to stop the processing of this record and move to the next record.
@Override
public V transform(K readOnlyKey, V value) {
try {
// do some logic
} catch(Exception e){
// process error - this might be unrecoverable.
dropRecord(); // this is what i would be looking for if possible
}
}
I could kill the thread and have a customUncaughtExceptionHandler reschedule the thread which would not commit the offset and therefor try to process the faulty record again.
Creating a wrapper for the objects passed would require to add a check in each proccessing step to see if the record is still valid.
Adding a.branch() before each processing step to would also require a decent amount of rework.
You can drop a message in a Transformer
simply by returning null
. See the Javadoc of Transformer#transform . So your example would be:
@Override
public V transform(K readOnlyKey, V value) {
try {
// do some logic
} catch(Exception e){
// process error - this might be unrecoverable.
return null;
}
}
Note, that you can do this currently only in a Transformer
, but not in a ValueTransformer
.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.