Airflow：自定義日期：適用於日期時間，但不適用於鍾擺

Question

需求：創建自定義日期 function 用於算子、DAG等

下面是 DAG 文件

有向無環圖

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta
from alerts.custom_date import strt_of_wk_strt_mon_dt, NEXT_DS_NODASH

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2020, 7, 8),
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG(dag_id = 'test_date',
        schedule_interval='@once',
        default_args=default_args)


def test(**kwargs):

    first_date = kwargs.get('execution_date', None)
    strt_wk_dt = kwargs.get('strt_wk_dt')
    next_ds_nodash = kwargs.get('next_ds_nodash')
    s3_key = kwargs.get('s3_key')

    print(f'EXECUTION DATE:{first_date}')
    print(f'STRT_WK_DT:{strt_wk_dt}')
    print(f'NEXT_DS_NODASH:{next_ds_nodash}')
    print(f'S#_KEY:{s3_key}')

with dag:
    execution_date = '{{ execution_date }}'
    next_ds_nodash = NEXT_DS_NODASH
    strt_wk_dt = strt_of_wk_strt_mon_dt()

    t1 = PythonOperator(
        task_id='show_template',
        python_callable=test,
        op_kwargs={'execution_date': execution_date,
                   'next_ds_nodash': next_ds_nodash,
                   'strt_wk_dt': strt_wk_dt,
                   's3_key':f'snowflakes/FEEDS/{strt_wk_dt}/abc_{strt_wk_dt}.csv'},
        provide_context=True)

使用 datetime package首先我嘗試使用 DateTime 庫，它工作正常，如下圖所示

下面是 cstm_date.py

from datetime import datetime, timedelta
import logging
logger = logging.getLogger(__name__)

NEXT_DS_NODASH = '{{ (execution_date + macros.timedelta(days=1)).strftime("%m%d%Y") }}'

def strt_of_wk_strt_mon_dt():
    return (datetime.today().date() - timedelta(days=datetime.today().weekday())).strftime('%Y_%m_%d')

output 正在按預期打印，如下所示

接下來，我嘗試使用擺庫，output 沒有打印日期值**擺包**

下面是 cstm_date.py

import pendulum
import logging
logger = logging.getLogger(__name__)

NEXT_DS_NODASH = '{{ (execution_date + macros.timedelta(days=1)).strftime("%m%d%Y") }}'


def strt_of_wk_strt_mon_dt():
    today = pendulum.now()
    return today.start_of('week').format('YYYY_MM_DD')

output 未打印STRT_WK_DT值

我錯過了什么？

Answer 1

您正在使用正確的datetime.strftime轉換指令，但未在使用pendulum的新 function 中使用字符串datetime轉換指令。

你的

def strt_of_wk_strt_mon_dt():
    today = pendulum.now()
    return today.start_of('week').format('YYYY_MM_DD')

故意的

def strt_of_wk_strt_mon_dt():
    today = pendulum.now()
    return today.start_of('week').format('%Y_%m_%d')