how to run feedconsumers and consumers multiple for kafka in docker?

Question

So I have this docker file and i want to run feed-consumers and consumers multiple times and i tried to do so. We have a node.js application for feed-consumers and consumer and pass user_levels to it.

I just want to ask is this the right approach?

FROM ubuntu:18.04
# Set Apt to noninteractive mode
ENV DEBIAN_FRONTEND noninteractive

# Install Helper Commands
ADD scripts/bin/* /usr/local/bin/

RUN chmod +x /usr/local/bin/*

RUN apt-install-and-clean curl \
    build-essential \
    git >> /dev/null 2>&1

RUN install-node-12.16.1

RUN mkdir -p /usr/src/app

COPY . /usr/src/app

WORKDIR /usr/src/app

#RUN yarn init-cache
#RUN yarn init-temp
#RUN yarn init-user

RUN yarn install

RUN yarn build

RUN node ./feedsconsumer/consumer.js user_level=0
RUN for i in {1..10}; do node ./feedsconsumer/consumer.js user_level=1; done
RUN for i in {1..20}; do node ./feedsconsumer/consumer.js user_level=2; done
RUN for i in {1..20}; do node ./feedsconsumer/consumer.js user_level=3; done
RUN for i in {1..30}; do node ./feedsconsumer/consumer.js user_level=4; done
RUN for i in {1..40}; do node ./feedsconsumer/consumer.js user_level=5; done

RUN for i in {1..10}; do node ./consumer/consumer.js; done

ENTRYPOINT ["tail", "-f", "/dev/null"]

Or is there any other way around?

Thanks

Answer 1

A container runs exactly one process. Your container's is

ENTRYPOINT ["tail", "-f", "/dev/null"]

This translates to "do absolutely nothing, in a way that's hard to override". I typically recommend using CMD over ENTRYPOINT , and the main container command shouldn't ever be an artificial "do nothing but keep the container running" command.

Before that, you're trying to RUN the process(es) that are the main container process. The RUN only happens during the image build phase, the running process(es) aren't persisted in the image, the build will block until these processes complete, and they can't connect to other containers or data stores. These are the lines you want to be the CMD .

A container only runs one processes, but you can run multiple containers off the same image. It's somewhat easier to add parameters by setting environment variables than by adjusting the command line (you have to replace the whole thing), so in your code look for process.env.USER_LEVEL . Also make sure the process stays as a foreground process and doesn't use a package to daemonize itself.

Then the final part of the Dockerfile just needs to set a default CMD that launches one copy of your application:

...
COPY package.json yarn.lock .
RUN yarn install
COPY . .
RUN yarn build
CMD node ./feedsconsumer/consumer.js

Now you can start a single container running this process

docker build -t my/consumer .
docker run -d --name consumer my/consumer

And you can start multiple containers to run the whole set of them

for user_level in `seq 5`; do
  for i in `seq 10`; do
    docker run -d \
      --name "feed-consumer-$user_level-$i" \
      -e "USER_LEVEL=$user_level" \
      my/consumer
  done
done
for i in `seq 10`; do
  docker run -d --name "consumer-$i" \
    my/consumer \
    node ./consumer/consumer.js
done

Notice this last invocation overrides the CMD to run the alternate script; this becomes a more contorted invocation if it needs to override ENTRYPOINT instead. ( docker run --entrypoint node my/consumer./consumer/consumer.js )

If you're looking forward to cluster environments like Kubernetes, it's often straightforward to run multiple identical copies of a container, which is what you're trying to do here. A Kubernetes Deployment object has a replicas: count, and you can kubectl scale deployment feed-consumer-5 --replicas=40 to change what's in the question, or potentially configure a HorizontalPodAutoscaler to set it dynamically based on the topic length (this last is involved, but possible and rewarding).