有谁知道如何抑制所有气流“信息”级日志，但不抑制特定于应用程序实施的日志？

Question

Airflow 1.10.1 has an attribute called "logging_level" that I believe is tied to the Python logging level. When the value is INFO or lower, the output logs are too verbose and unnecessary in deployments.

Rather, I want to be able to log just airflow framework errors, and everything I want my application to log. Then I cut down on the logging to something minimal, most just in the context of the application, and only keep airflow framework/execution errors.

In a particular PythonOperator I wrote at 5 different levels of log to see what happens to them when I modify the airflow.cfg logging_level.

 logging.debug('******************* HELLO debug *******************') logging.info('******************* HELLO info *******************') logging.warning('******************* HELLO warning *******************') logging.error('******************* HELLO error *******************') logging.critical('******************* HELLO critical *******************') 

The idea being that by changing the airflow.cfg attribute for logging_level from debug to info to warning, I can see less and less of the airflow logs, and just leave the application specific logs I want.

Step 1: logging_level = DEBUG

Here's the log from the task that has logs at all level from debug upward.

Step 2: logging_level = INFO

As expected, the logs do not include debug level messages.

Step 3: logging_level = WARNING

When we go up from INFO to WARNING, the file is empty. I was expecting the warning, error, and critical messages in the file and the rest suppressed from Airflow since the log did not contain anything from airflow at the level above INFO.

Step 4: logging_level = ERROR

The same problem here again, the file is empty. I expected to get the error and critical messages, but the file is empty.

Note, in the last two screenshots, it's not that the path is invalid, but Airflow just displays the path to the file it seems in the absence of any content in the log file.

So my question is: 1) Is this just an Airflow bug? 2) Am I not using this properly? Or do I need to do something else in order to suppress Airflow level logs from INFO and below in production, and just keep my application specific logs?

Answer 1

If you notice in your log screenshots your log message are actually wrapped in an info log. If you want to actually change the log level within the task log and not wrap it you can pull the log off of the task instance (from the **kwargs) and use it directly as opposed to generically calling logging.warning(). Here is an example:

def your_python_callable(**kwargs):
    log = kwargs["ti"].log
    log.warning("******HELLO Debug******")