Using Kafka with Netflix Conductor

Question

I want to know if there is an easy way to connect Kafka and Netflix Conductor (instead of SQS)? At now, it seems to work only with Amazon SQS. Moreover, it seems that it is only possible to make one action by task. Is there a way to execute more than one action by task?

Thanks in advance,

Answer 1

To add Kafka support to Netflix Conductor, you will need to

• Create a module in contribs that extends AbstractModule (Add an entry in your server.properties against conductor.additional.modules property)
• Implement ObservableQueue for kafka producer and consumer operations.
• Implement EventQueueProvider just like SQS implementation
• Add properties for your kafka initialization in server.properties

kafka.producer.bootstrap.servers=host1:port1;host2:port2
kafka.producer.key.serializer=org.apache.kafka.common.serialization.StringSerializer
kafka.producer.value.serializer=org.apache.kafka.common.serialization.StringSerializer

kafka.consumer.bootstrap.servers=host1:port1;host2:port2
kafka.consumer.key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
kafka.consumer.value.deserializer=org.apache.kafka.common.serialization.StringDeserializer

• Add kafka library support to the project by updating the build.gradle of contribs modules.

Refer to the below PR link for implementation (Authored by preeth-gopalakrishnan)

https://github.com/Netflix/conductor/pull/672

(If you don't find the PR, assume it is merged to the master)