AWS Lambda read-only file system error failed to create directory with Docker image

Question

Problem

Docker image compiles successfully, however fails when ran from Lambda because of its read only file system.

Summary

Luminati-proxy has a docker integration for their proxy manager. I copied over their docker file and appended it to my own docker file for pushing out a script to AWS Lambda. The building of the docker image was successful , but when pushed off to Lambda, it failed because of a read only file system error :

Failed to create directory /home/sbx_user1051/proxy_manager: [code=EROFS] Error: EROFS: read-only file system, mkdir '/home/sbx_user1051'
2022-02-28 19:37:22.049 FILE (8): Failed to create directory /home/sbx_user1051/proxy_manager: [code=EROFS] Error: EROFS: read-only file system, mkdir '/home/sbx_user1051' 

Analysis

Upon examining the trackback, the error is focused on the proxy_manager installation and fails with directory changes (mkdir, mk_work_dir...). These changes are made within the.js files of the GitHub which is pulled from the docker file as the proxy_manager installation. Obviously the only mutable directory on Lambda is the /tmp directory, but is there a workaround for getting this set up without resorting to putting everything under the /tmp directory as it wipes itself upon runtime? Reinstalling a proxy_manager each run is not at all ideal ...

Answer?

Could this be as simple as setting environment stipulations such as:

ENV PATH=...
ENV LD_LIBRARY_PATH=...

If so, I how should they be configured? I am adding the docker file below for quick reference:

FROM node:14.18.1
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
    && apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf \
      --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

USER root
RUN npm config set user root
RUN npm install -g npm@8.1.3
RUN npm install -g @luminati-io/luminati-proxy
ENV DOCKER 1
CMD ["luminati", "--help"]

I appreciate the insight!

Answer 1

TL;DR:

• You should instead leverage an S3 bucket to store, read and modify any file. All lambdas and microservices. In general, should always be treated as stateless
• All Luminati-proxy functionality comes prebuilt within amazon lambdas and API Gateway
• Lambda functions are not meant to run long-running processes as they are limited to 15 minutes maximum so the design of the container that you are trying to run in lambdas has to have AWS serverless architecture considerations in its design

Explanation:

According to thedocumentation of AWS Lambda functions:

The container image must be able to run on a read-only file system. Your function code can access a writable /tmp directory with 512 MB of storage.

Since containers based on Linux based images are already supposed to have a folder called /tmp you should pretty much be able to access that folder any time from your code to read ( remember, read-only FS)

If you are looking to store content amazon's solution for that is for you to have any content created and manage over an S3 bucket, buckets are as easy to use as if you read a file locally but will remain accessible after the lambda instance finishes the workload

Please refer to Read file from aws s3 bucket using node fs and Upload a file to Amazon S3 with NodeJS for more details on how to use an S3 bucket. There are plenty of ways to achieve it regardless of the language been used.

This is all based on a best practice promoted by AWS over their platform. Where lambdas remain stateless

Answer 2

AWS Lambda provides /tmp folder for users to write files on lambda, as I don't about your question context but hope this help. You can write files to AWS Lambda at /tmp folder eg. I want to create a file demo.txt at runtime/programmatically using AWS lambda, then i can write the file to /txt/demo.txt