FileNotFoundError Python-Flask/Docker in Kubernetes Cluster

Question

I am having problems with opening files in my Flask app, running in Docker container in a Kubernetes cluster.

Whenever I try to open a file i directory I am getting a FileNotFoundError . S3FS Object Storage is provisioned and the PersistentVolumeclaim is successfully mounted to my pod, so no issue there.

However, when trying to read a file from the mounted storage I get the said error. Here is how I read the file. The current_dir is /var/www

current_dir = os.path.dirname(__file__)
abs_path = os.path.join(current_dir, path)
with open(abs_path, "rb") as file:
    bin_data = file.read()

Additonally I have also tried all of the variants below, but none seem to work:

data_path = pathlib.Path('/var/www/uploads/')
abs_path = data_path / path_id

data_path = pathlib.Path('uploads/')
abs_path = data_path / path_id

abs_path = current_dir + '/uploads' + path_id

tmp = str('/uploads/' + path_id).split('/')
abs_path = os.path.join(os.path.dirname(__file__), *tmp)

The error I am getting is following:

[2020-04-02 18:16:49,359] ERROR in app: Exception on /internal/tusd [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/var/www/api_controller.py", line 95, in tusd_handler
    hexdigest = hash_file(str(upload_id), hash_type)
  File "/var/www/file_hasher.py", line 17, in hash_file
    with open(abs_path, "rb") as file:
FileNotFoundError: [Errno 2] No such file or directory: '/var/www/uploads/f7f1c686fbc40649879e8aee7f529c96'

However, when I exec -it into the container. The file, among others is present.

$ kubectl exec hasher-savoury-695669b858-nwg5j -- ls /var/www/uploads

f7f1c686fbc40649879e8aee7f529c96
f7f1c686fbc40649879e8aee7f529c96.info

Here is the Dockerfile I have been using. The commented fields are options I have tried out in order to get it working.

FROM python:3.7-alpine as base

ENV GROUP_ID=1000 \
    USER_ID=1000

RUN apk upgrade && apk update \
    && apk --no-cache add openssl-dev openssl
RUN apk --no-cache --update add --virtual build-dependencies gcc g++ make libffi-dev

RUN mkdir /var/www && mkdir /var/www/uploads
COPY . /var/www
WORKDIR /var/www

RUN pip install --upgrade pip && pip install -r requirements.txt && apk del build-dependencies
RUN pip install gunicorn

#RUN addgroup -g $GROUP_ID savoury
#RUN adduser -D -u $USER_ID -G savoury savoury -s /bin/sh \
#    && chown savoury:savoury -R var/www/

#USER savoury

EXPOSE 5080

ENTRYPOINT ["gunicorn"]
CMD ["-w", "4", "--bind", "0.0.0.0:5080", "wsgi"]

I have tried reading these files in Java an have succesfully read them. I have also tried to ´´´exec´´´ into the container and was able to perform this easily in the python console. I have no idea why the script above is not working.

Any help is greatly appreciated!

Answer 1

The reason behind the error was in fact that the file was not found. It did not exist during time of reading because of improper timing of async events with the uploads server. The fault was undiscovered due to poor logging. Lesson: Always log and monitor.