在MongoSink - alpakka mongo连接器响应后，向kafka消费者致敬

Question

I am using alpakka connector to consume packets from Kafka and insert them into Mongo db. I was trying to commit the offset after getting response from Mongo db but couldn't find anything regarding the same. How can I make sure that offset will be committed only after the packet successfully inserted to the Mongodb?

I am using Consumer.CommittableSource as a source and MongoSink as a sink and running the stream using RunnableGraph. Please refer the code for more clarification.

Source:

    public Source<ConsumerMessage.CommittableMessage<String, String>, Consumer.Control> source() {
        return Consumer.committableSource(consumerSettings, subscription);
    }

Flow:

   public Flow<ConsumerMessage.CommittableMessage, String, NotUsed> transformation() {
        return Flow.of(ConsumerMessage.CommittableMessage.class).map(i -> i.record().value().toString());
    }

Sink:

    public Sink<String, CompletionStage<Done>> sink() {
        return MongoSink.insertOne(mongoCollection);
    }

Graph:

        RunnableGraph graph = RunnableGraph.fromGraph(GraphDSL.create(sink(), (builder, s) -> {
            builder.from(builder.add(source()).out()).via(builder.add(transformation())).to(s);
            return ClosedShape.getInstance();
        }));
        graph.run(ActorMaterializer.create(actorSystem));

EDIT :

Using PassThroughFlow, insertion to Mongo is working and it is not giving any exception or error but still not able to commit the packet. transformationCommit() function has never been called.

Updated Flow :

    public Flow<String, String, NotUsed> transformationMongo() {
        LOGGER.info("Insert into Mongo");
        return  MongoFlow.insertOne(connection.getDbConnection());
    }

public Flow<ConsumerMessage.CommittableMessage, ConsumerMessage.CommittableOffset, NotUsed> transformationCommit() {
        return Flow.of(ConsumerMessage.CommittableMessage.class).map(i -> i.committableOffset());
    }

Sink:

    public Sink<ConsumerMessage.CommittableOffset, CompletionStage<Done>> sinkCommit() {
        CommitterSettings committerSettings = CommitterSettings.create(actorSystem);
        return Committer.sink(committerSettings);
    }

PassThroughFlow:

public class PassThroughFlow {
    public Graph<FlowShape<ConsumerMessage.CommittableMessage, ConsumerMessage.CommittableMessage>, NotUsed> apply(Flow<ConsumerMessage.CommittableMessage, String, NotUsed> flow) {
        return Flow.fromGraph(GraphDSL.create(builder -> {
            UniformFanOutShape broadcast = builder.add(Broadcast.create(2));
            FanInShape2 zip = builder.add(ZipWith.create((left, right) -> Keep.right()));
            builder.from(broadcast.out(0)).via(builder.add(flow)).toInlet(zip.in0());
            builder.from(broadcast.out(1)).toInlet(zip.in1());
            return FlowShape.apply(broadcast.in(), zip.out());
        }));
    }
}

Graph:

RunnableGraph graph = RunnableGraph.fromGraph(GraphDSL.create(sinkCommit(), (builder, s) -> {
            builder.from(builder.add(source()).out()).via(builder.add(passThroughFlow.apply(transformation().via(transformationMongo())))).via(builder.add(transformationCommit())).to(sales);
            return ClosedShape.getInstance();
        }));
        graph.run(ActorMaterializer.create(actorSystem));

//Insertion to mongo is working but still the packet not committed. transformationCommit() function has never been called.

Answer 1

而不是MongoSink ，使用MongoFlow ，它允许您继续流，然后您将能够提交偏移。