Pushing to Diego: Cannot write: No space left on device

Question

Our application is one of the few left running on DEA. On DEA we were able to use a specific custom buildbpack:

https://github.com/ihuston/python-conda-buildpack

Now that we have to move on Diego runtime, we run out of space while pushing the app. I believe the disk space is only required during staging, because quite a few libraries are coming with the buildpack and have to be built (we need the whole scientific python stack, which is all included in the above buildpack).

The build script outputs everything fine, except that the app cannot start. The logs then show:

2016-10-13T19:10:42.29+0200 [CELL/0]     ERR Copying into the container failed: stream-in: nstar: error streaming in: exit status 2. Output: tar: ./app/.conda/pkgs/cache/db552c1e.json: Wrote only 8704 of 10240 bytes

and further many files:

2016-10-13T19:10:42.29+0200 [CELL/0]     ERR tar: ./app/.conda/pkgs/cache/9779607c273dc0786bd972b4cb308b58.png: Cannot write: No space left on device

and then

2016-10-13T20:16:48.30+0200 [API/0]      OUT App instance exited with guid b2f4a1be-aeda-44fa-87bc-9871f432062d payload: {"instance"=>"", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"Copying into the container failed", "crash_count"=>14, "crash_timestamp"=>1476382608296511944, "version"=>"ca10412e-717a-413b-875a-535f8c3f7be4"}

When trying to add more disk quota (above 1G) there is an error:

Server error, status code: 400, error code: 100001, message: The app is invalid: disk_quota too much disk requested (must be less than 1024)

Is there a way to give a bit more space? At least for the build process?

Answer 1

You can use a .cfignore file just like a .gitignore file to exclude any unneeded files from being cf push ed. Maybe if you really only push what is necessary, the disk space could be sufficient.

https://docs.developer.swisscom.com/devguide/deploy-apps/prepare-to-deploy.html#exclude

Answer 2

The conda installer from https://github.com/ihuston/python-conda-buildpack installs by default with the Intel MKL library. Now this is usually a good thing, but seemingly uses too much space and thus cannot be deployed.

I adapted the buildpack and added to the line

        $CONDA_BIN/conda install --yes --quiet --file "$BUILD_DIR/conda_requirements.txt"

The flag nomkl

        $CONDA_BIN/conda install nomkl --yes --quiet --file "$BUILD_DIR/conda_requirements.txt"

As described in continuums blog post here:

https://www.continuum.io/blog/developer-blog/anaconda-25-release-now-mkl-optimizations

This will then use OpenBLAS instead and results in a much smaller droplet (175M instead of 330MB) and the deployment can successfully finish.