How do i have a JAVA_HOME set using a Dockerfile and python?

Question

I am trying to set up a Dockerfile for my project and am unsure how to set a JAVA_HOME within the container.

FROM python:3.6
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
COPY . /app
# Install any needed packages specified in requirements.txt
RUN pip install --trusted-host pypi.python.org -r requirements.txt
# Define environment variable
ENV NAME Netflow
# Run netflow.py
CMD ["python", "netflow.py"]

In the requirements.txt I have...

numpy
pandas
kafka
pyspark
log

My netflow.py file is...

import pandas, math, re, log
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext


conf = SparkConf().setAppName("building a warehouse")
sc = SparkContext(conf=conf)
df=pandas.read_csv(r'TestDataSet.csv') 

The output in the terminal after trying to run it is....

JAVA_HOME is not set
Traceback (most recent call last):
  File "netflow.py", line 7, in <module>
    sc = SparkContext(conf=conf)
  File "/usr/local/lib/python3.6/site-packages/pyspark/context.py", line 115, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/usr/local/lib/python3.6/site-packages/pyspark/context.py", line 298, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/usr/local/lib/python3.6/site-packages/pyspark/java_gateway.py", line 94, in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")

I have been looking for a solution but none have worked so far.

I have tried

ENV JAVA_HOME /Library/Java/JavaVirtualMachines/openjdk-11.jdk/Contents/Home

and I have tried using a separate command

docker run -e "JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk-11.jdk/Contents/Home" project env

I am still getting the same error

Answer 1

You need to actually install Java inside your container, but I would suggest rather finding a Pyspark docker image, or adding Python to the Openjdk images so that you don't need to mess with too many environment variables

More specifically, JAVA_HOME=/Library/Java/JavaVirtualMachines is a only available as a path to your Mac, and shouldn't be expected to work inside a Linux container

However, it's not clear why you need Pyspark when numpy is the only thing actually reading your data

Answer 2

To set environment variables, you can declare them in your dockerfile like so:

ENV JAVA_HOME="foo"

or

ENV JAVA_HOME foo

In fact, you already set an environment variable in the example you posted.

See documentation for more details.