[英]How to mount a volume to a Kubernetes Pod running a docker container from the Airflow host that triggers it using the KubernetesPodOperator
I have a DAG in airflow that uses the KubernetesPodOperator and I am trying to get some files that are generated by the container running in the pod back to the airflow host.我在气流中有一个 DAG,它使用KubernetesPodOperator ,我正在尝试将由在 pod 中运行的容器生成的一些文件返回到气流主机。 For development my host is a Docker container running an airflow image with a docker-desktop K8s cluster and for production I am using an AWS EC2 box with EKS.
对于开发,我的主机是一个 Docker 容器,它运行带有 docker-desktop K8s 集群的气流映像,对于生产,我使用的是带有 EKS 的 AWS EC2 盒。
volume_mount = VolumeMount('dbt-home',
mount_path=<CONTAINER_DIR>,
sub_path=None,
read_only=False)
volume_config= {
'hostPath':
{'path': <HOST_DIR>, 'type': 'DirectoryOrCreate'}
}
volume = Volume(name="dbt-home", configs=volume_config)
dbt_run = KubernetesPodOperator(
namespace='default',
image=MY_IMAGE>,
cmds=["bash", "-cx"],
arguments=[command],
env_vars=MY_ENVIRONMENT,
volumes=[volume],
volume_mounts=[volume_mount],
name="test-run",
task_id="test-run-task",
config_file=config_file,
get_logs=True,
reattach_on_restart=True,
dag=dag
)
I tried using the hostPath type for the volume but i think that it refers to the host of the pod.我尝试对卷使用 hostPath 类型,但我认为它指的是 pod 的主机。 I looked in the kubernetes documentation around volumes where I found the EmptyDir one which didnt work out either.
我查看了有关卷的 kubernetes 文档,我发现 EmptyDir 也没有解决。
Based on your comment, you are asking how one task run in a pod can complete and write logs to a location that another task run in a pod can read when it starts.根据您的评论,您是在询问在 pod 中运行的一个任务如何完成并将日志写入一个位置,以便在 pod 中运行的另一个任务在启动时可以读取。 It seems like you could do a few things.
似乎你可以做一些事情。
kubectl get logs
(ie- put kubectl into your task image and permission its service account to get the logs of pods in that namespace) or use the Kubernetes python API to get the logs.kubectl get logs
完成的前一个 pod 的kubectl get logs
(即将 kubectl 放入您的任务映像并允许其服务帐户获取该命名空间中的 pod 日志)或使用 Kubernetes python API 来获取日志。 I am not sure if you are looking for a more airflow native way of doing this, but those are ideas that come to mind that would solve your problem.我不确定您是否正在寻找一种更气流原生的方式来做到这一点,但这些想法可以解决您的问题。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.