Trouble converting Docker to Singularity: "Function not implemented" in Singularity, but works fine in Docker

Question

I have an Ubuntu docker container that works perfectly fine as is. I have a custom binary inside that executes and returns as expected. Because of security reasons, I cannot use docker for automated testing. I created a docker archive and then I load a singularity container from this docker archive. The binary that I need to run fails with the following error:

MyBinary::BinaryNameSpace::BinaryFunction[FATAL]: boost::filesystem::status: Function not implemented: "/var/tmp/username"

When I run $ldd <binary_path>, I see that a boost filesystem binary was linked. I am not sure why the binary is unable to find the status function...

So far, I have used a tool called ermine to turn the dynamically linked binary into a static binary

I still got the same error, which I found very strange.

Any suggestions on directions to look next are very appreciated. Thank you.

Answer 1

Both /var/tmp and /tmp are silently automounted by default . If anything was added to /var/tmp during singularity build or in the source docker image, it will be hidden when the host's /var/tmp is mounted over it.

You can disable the automounts individually when you run a singularity command, which is probably what you want to do first to check that it is the source of the problem ( eg, singularity run --no-mount tmp... ). I'd also recommend using --writable-tmpfs or manually mounting -B /tmp to make sure that there is somewhere writable for any temp files. You are likely to get an error about a read-only filesystem if not.

The host OS environment can also cause problems in unexpected ways that are hard to debug. I recommend using --cleanenv as a general practice to minimize this.

Answer 2

The culprit was an outdated Linux kernel. The containers still use the host's kernel.

On Docker, I was using Kernel 5.4.x and the computer that runs the singularity container runs 3.10.x

There are instructions in the binary which are not supported on 3.10.x

There is no fix for now except running the automated tests on a different computer with a newer kernel.