Install apt-get dependencies on Google Dataflow with Beam Java SDK

Question

We are currently trying to get OpenCV running in a Java job on Google Cloud Dataflow. Unfortunately, we can not replace the Docker container that Dataflow is using with one that has OpenCV installed. ( See other question ) If we used the Python SDK there is an option to specify a setup.py file that can be used to invoke apt-get . Is there something similar for jobs created with the Java SDK?

thanks for you help!

Answer 1

I came up with a solution, but there might be a more elegant way to do this.

@Setup
public void setupDoFn() {
    ProcessBuilder pb = new ProcessBuilder("apt-get", "install", "-y", "libopencv-dev");
    try {
        Process p = pb.start();
        String line;
        BufferedReader input =
                new BufferedReader
                        (new InputStreamReader(p.getInputStream()));
        while ((line = input.readLine()) != null) {
            logger.debug("Apt-get: " + line);
        }
        input.close();
        // Initialize the OpenCV Libarary
        nu.pattern.OpenCV.loadLibrary();
    } catch (IOException e) {
        e.printStackTrace();
        // If we could not install OpenCV, we have to terminate the stream
        System.exit(-1);
    }

}

Answer 2

Cloud Dataflow now supports Custom Containers, which can have other libraries (eg OpenCV) installed. See this page for more information.