How to run Kafka from Jenkins Pipeline using Groovy and Kubernetes plugin?

Question

I couldn't find such a specific command around the internet so I kindly ask for your help with this one :)

Context

I have defined a podTemplate with a few containers, by using the containerTemplate methods:

• ubuntu:trusty (14.04 LTS)
• postgres:9.6
• and finally, wurstmeister/kafka:latest

Doing some Groovy coding in Pipeline, I install several dependencies into my ubuntu:trusty container, such as latest Git, Golang 1.9, etc., and I also checkout my project from Github.

After all that dependencies are dealt with, I manage to compile, run migrations (which means Postgres is up and running and my app is connected to it), and spin up my app just fine until it complains that Kafka is not running because it couldn't connect to any broker.

Debugging sessions

After some debug sessions I have ps aux 'ed each and every container to make sure all the services I needed were running in their respective containers, such as:

    container(postgres) {
        sh 'ps aux'  # Show Postgres, as expected
    }

    container(linux) {
        sh 'ps aux | grep post'  # Does not show  Postgres, as expected
        sh 'ps aux | grep kafka' # Does not show Kafka, as expected
    }

    container(kafka) {
        sh 'ps aux' # Does NOT show any Kafka running
    }

I have also exported KAFKA_ADVERTISED_HOST_NAME var to 127.0.0.1 as explained in the image docs, without success, with the following code:

    containerTemplate(
            name: kafka,
            image: 'wurstmeister/kafka:latest',
            ttyEnabled: true,
            command: 'cat',
            envVars: [
                    envVar(key: 'KAFKA_ADVERTISED_HOST_NAME', value: '127.0.0.1'),
                    envVar(key: 'KAFKA_AUTO_CREATE_TOPICS_ENABLE', value: 'true'),
            ]
    )

Questions

This image documentation details https://hub.docker.com/r/wurstmeister/kafka/ is explicit about starting a Kafka cluster with docker-compose up -d

1) How do I actually do that with this Kubernetes plugin + Docker + Groovy + Pipeline combo in Jenkins?

2) Do I actually need to do that? Postgres image docs ( https://hub.docker.com/_/postgres/ ) also mentions about running the instance with docker run , but I didn't need to do that at all, which makes me think that containerTemplate is probably doing it automatically. So why is it not doing this for the Kafka container?

Thanks!

Answer 1

So... problem is with this image, and way how kubernetes works with them. Kafka does not start because you override dockers CMD with command:'cat' which causes start-kafka.sh to never run. Because of above I suggest using different image. Below template worked for me.

containerTemplate(
    name: 'kafka',
    image: 'quay.io/jamftest/now-kafka-all-in-one:1.1.0.B',
    resourceRequestMemory: '500Mi',
    ttyEnabled: true,
    ports: [
        portMapping(name: 'zookeeper', containerPort: 2181, hostPort: 2181),
        portMapping(name: 'kafka', containerPort: 9092, hostPort: 9092)
    ],
    command: 'supervisord -n',
    envVars: [
        containerEnvVar(key: 'ADVERTISED_HOST', value: 'localhost')
    ]
),