如果我们有 airflow 用于编排，为什么我们需要 TFX

Question

I still don't get why we need TFX. TFX will convert your defined pipeline to Airflow DAG and run it on airflow, I could just write my pipelines in python and use Airflow's PythonOperator to build a pipeline directly right? why bother learning another wrapper on top of it? What else TFX offers that cannot be done by just using airflow+TF+Spark/Beam

Answer 1

I could just write my pipelines in python and use Airflow's PythonOperator to build a pipeline directly right?

You can! Depending on how you define a pipeline of course.

Here is the definition of TFX, from it's guide :

" TFX is a Google-production-scale machine learning (ML) platform based on TensorFlow. It provides a configuration framework and shared libraries to integrate common components needed to define, launch, and monitor your machine learning system. "

And to make a Production ML System

according to engineers at Tensorflow .

So, if you can define a whole system where you are able cover all these steps in Airflow DAG's, sure you don't need TFX.

---

PS :

It comes down to the problem you are trying to solve. Here are some questions to think about.

• Do you have the data needed at hand, is it valuable?

• Do you need to adjust it before giving it to a model?

• Which model should you use?

• Are you going to re-train the model as you get new data? If so what is the period of this process should be?

• As you are doing inference - or serving your model - how are you going to use the predicted results?

• What is your threshold for evaluating the success of your service? What metrics should you use?

To learn more, you can check here .