DBT运行（或特定模型）完成后如何运行python代码？

Question

我希望能够运行一个 ad-hoc python 脚本，该脚本可以访问和运行由 dbt 运行计算的模型的分析，是否有任何最佳实践？

Answer 1

For production, I'd recommend an orchestration layer such as apache airflow.

See this blog post to get started, but essentially you'll have an orchestration DAG (note - not a dbt DAG) that does something like:

dbt run <with args> -> your python code

Fair warning, though, this can add a bit of complexity to your project.

I suppose you could get a similar effect with a CI/CD tool like github actions or circleCI

Answer 2

We recently built a tool that could that caters very much to this scenario. It leverages the ease of referencing tables from dbt in Python-land. It's called fal .

The idea is that you would define the python scripts you would like to run after your dbt models are run:

# schema.yml
models:
- name: iris
  meta:
    owner: "@matteo"
    fal:
      scripts:
        - "notify.py"

And then the file notify.py is called if the iris model was run in the last dbt run :

# notify.py
import os
from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError

CHANNEL_ID = os.getenv("SLACK_BOT_CHANNEL")
SLACK_TOKEN = os.getenv("SLACK_BOT_TOKEN")

client = WebClient(token=SLACK_TOKEN)
message_text = f"""Model: {context.current_model.name}
Status: {context.current_model.status}
Owner: {context.current_model.meta['owner']}"""


try:
    response = client.chat_postMessage(
        channel=CHANNEL_ID,
        text=message_text
    )
except SlackApiError as e:
    assert e.response["error"]

Each script is ran with a reference to the current model for which it is running in a context variable.

---

To start using fal, just pip install fal and start writing your python scripts.