Running Kafka Consumers that interact with a Flask application

Question

hoping to get some help on sort of a general architectural question.

I have a Flask Web Application (let's call it the people-app ) that runs in a Docker container. It's deployed to Amazon ECS, and appears to be running fine. people-app exposes an HTTP API in which authorized users can interact using a token, as well as some traditional web-app pages.

What I'd like to do, is implement a Kafka Consumer Group which meant to consume from a topic related to the Flask application (ie people-topic ), and on consumption, interact with the Flask application (maybe creating model instances and saving them to the database, similar to how the HTTP API would handle an HTTP request). I haven't really been able to find any implementation examples anywhere, especially in python, but I'd say I'm trying to achieve something similar to the eventuate framework from Chris Richardson.

I guess my question boils down to; How would I implement something like this? If Kafka Consumers run in a continuous loop, I assume that they should be run in their own, separate process. This appears to create issues with Docker, as running multiple process in a container effectively removes the ability for the container to efficiently communication with the application(s).

Should I be running separate containers? If so, how would these two containers then communicate? Am I missing something? Am I completely on the wrong path here?

Any help will be greatly appreciated, even if it's something small to get me on the right track here. Thank you in advance.

Answer 1

I would suggest to use separate containers for your flask app and kafka app. This follows good software design principle of single responsibility.

You need not learn java, you can code in python, please check the link below https://towardsdatascience.com/kafka-python-explained-in-10-lines-of-code-800e3e07dad1

You can check AWS documents for container networking in ECS https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html[2]

Please check this article on how can you create multiple tasks or service. You can use either awsvpc or bridge network. In awsvpc mode each of your container will have own private ip address and can attach security group. With bridge network, you can map container port to host and containers can access each other by localhost:port . https://www.freecodecamp.org/news/amazon-ecs-terms-and-architecture-807d8c4960fd/ https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html

You can also check AWS Farget for your container runtime