Airflow 微服务架构

Question

My current platform architecture has a microservice for downloading/collecting data, a microservice for ETL and another microservice to process some complex SQL scripts.

I want to use Airflow to schedule and monitor workflows. I tried it and it worked very nice. However, I have to put all functionalities as tasks into one container of Airflow; and this doesn't follow current microservice architecture. What I wanted is to use Airflow as a scheduler and communicate with other microservices.

I want to ask: What is the best way to use Airflow with microservices? Should I use tasks in DAGs to communicate with microservices (publish messages and microservices will subscribe)?

DAG can be described as below. Please note that there're other tasks like validation after downloading data but I've just simplified it. DAG

Answer 1

A workflow engine like Apache Airflow and an architectural paradigm like microservices are inherently antithetical. Both are perfectly legitimate, both valuable, and both have pros & cons, but they are two completely different approaches to building distributed systems.

You hit on it yourself in your comment:

..if I gather all functionalities of those microservices into one under airflow container, that would be monolithic.

The microservices principle being violated here is referred to as " smart endpoints, dumb pipes ".

The idea being 'Smart Microservice A' should communicate with 'Smart Microservice B' (be that directly or indirectly) and that you should not have 'Dumb Microsevice A' and 'Dumb Microservice B' wired together by 'Smart Workflow Service'.

The latter is more of an Enterprise Service Bus (ESB) or a Service Oriented Architecture (SOA) design.

Again, ESBs and SOAs have their place, but a microservices architecture is a different kind of architecture and is inherently incompatible.