Apache Oozie 可以運行 docker 容器嗎？

Question

目前正在比較基於 DAG 的工作流工具，如 Airflow 和 Luigi，用於調度通用 docker 容器以及 Spark 作業。

Apache Oozie 能否通過其shell操作運行通用 Docker 容器？ 還是 Oozie 嚴格適用於 Pig 和 Hive 等 Hadoop 工具？

Oozie 與 Hadoop 堆棧的其余部分集成，支持多種開箱即用的 Hadoop 作業類型（例如 Java map-reduce、Streaming map-reduce、Pig、Hive、Sqoop 和 Distcp）以及系統特定作業（例如Java 程序和 shell 腳本）。

Answer 1

我嘗試通過 Shell 操作運行 Docker 容器並且它正在運行。 由於 Shell 動作可以在集群的任何節點上執行，所以 Docker 必須安裝在任何節點上。

從 Hue 創建的工作流.xml

<workflow-app name="Test docker" xmlns="uri:oozie:workflow:0.5">
    <start to="shell-5c29"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="shell-5c29">
        <shell xmlns="uri:oozie:shell-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <exec>test_docker.sh</exec>
            <file>/test_docker.sh#test_docker.sh</file>
        </shell>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>

test_docker.sh

docker run hello-world > output.txt
hdfs dfs -put -f output.txt /output.txt
echo 'done'

生成的 output.txt 的內容

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub (amd64)
 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/