[英]How to run a command in another pod in another namespace from an Airflow worker in a Kubernetes cluster
I want to run a papermill
command in a pod and get the errors (if any). 我想在pod中运行papermill
命令并获取错误(如果有的话)。 This has to be done from an Airflow worker using the bash operator
i Airflow. 这必须由使用bash operator
i Airflow的Airflow工作人员完成。
I have tried running the command as needed, I have tried running multiple bash commands from my own worker, commands that were meant to be executed inside the worker. 我已经尝试过根据需要运行命令,我尝试从我自己的worker中运行多个bash命令,这些命令是在worker中执行的。 But this just gives me the idea that something like this is possible. 但这只是让我觉得这样的事情是可能的。 Although, if there is any better approach that would solve this , I am all ears. 虽然,如果有任何更好的方法可以解决这个问题 ,我很满意。
The bash command to be run is: 要运行的bash命令是:
papermill check1.ipynb check1_output.ipynb -p params '{"aviral":"srivastava", "apoorv":"srivastava"}'
Manually, I exec
in my pod and then run this command. 手动,我exec
我的pod然后运行此命令。 I want to make this a bash command in airflow's dag. 我想在airflow的dag中将它作为bash命令。
To use kubectl
from within a pod
you need to have binaries available in your pod
. 要在广告pod
使用kubectl
,您需要在广告pod
提供二进制文件。
You can either Create a Docker image with installation of kubectl
. 您可以通过安装kubectl
来创建Docker镜像 。
I think the Dockerfile might look like this: 我认为Dockerfile可能如下所示:
FROM ubuntu:14.04
# Install.
RUN \
sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
apt-get update && \
apt-get -y upgrade && \
apt-get install -y build-essential && \
apt-get install -y software-properties-common && \
apt-get install -y byobu curl git htop man unzip vim wget && \
rm -rf /var/lib/apt/lists/* && \
# Installing kubectl using native package management
apt-get install -y apt-transport-https && \
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
echo 'deb https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee -a /etc/apt/sources.list.d/kubernetes.list && \
apt-get update && \
apt-get install -y kubectl
# Add files.
ADD root/.bashrc /root/.bashrc
ADD root/.gitconfig /root/.gitconfig
ADD root/.scripts /root/.scripts
# Set environment variables.
ENV HOME /root
# Define working directory.
WORKDIR /root
# Define default command.
CMD ["bash"]
Or Define a Command and Arguments for a Container and use it to install kubectl
when pod
is starting. 或者为Container定义一个命令和参数,并在pod
启动时使用它来安装kubectl
。
If you want to use it from within Kubernetes Deployment the part of .yaml
might look like this: 如果你想从Kubernetes部署中使用它的一部分.yaml
可能是这样的:
...
spec:
containers:
- name: ubuntu-with-kubectl
image: ubuntu
command: ["/bin/sh","-c"]
args:
- apt-get update && apt-get install -y apt-transport-https;
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -;
echo 'deb https://apt.kubernetes.io/ kubernetes-xenial main' | tee -a /etc/apt/sources.list.d/kubernetes.list;
apt-get update;
apt-get install -y kubectl;
...
You can also kubectl exec -it <pod_name> bash
into the pod and install is manually using this guide Install and Set Up kubectl , but it will be gone on container restart . 你也可以kubectl exec -it <pod_name> bash
进入pod并使用本指南手动安装安装和设置kubectl , 但它将在容器重启时消失 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.