Tomcat 9 running on docker - Cannot find /usr/local/tomcat/bin/setclasspath.sh

Question

I'm running a tomcat (tomcat:9-jre11) on docker, when launching it, it logs the following, then crashes:

Cannot find /usr/local/tomcat/bin/setclasspath.sh
This file is needed to run this program

My first issue was actually getting inside the container because I can't use docker exec on a crashed container, but I managed it by setting an entry point as /bin/bash in Rancher.

Now setclasspath.sh is very much in the /usr/local/tomcat/bin/ inside the container. It previously had all read and execution rights, I've set it to 777 just to be sure, still have the same issue. Same goes with changing the owner (tomcat seems to be using root, even if I launch the catalina.sh manually on another user, having changed the file owner). I used the heavy handed approch and set the whole damn folder as 777, and still the same:

drwxrwxrwx 1 root root   4096 Jun 29 14:53 .
drwxr-xr-x 1 root root   4096 Jun 29 14:31 ..
-rwxrwxrwx 1 root root  34699 Jun  2 21:08 bootstrap.jar
-rwxrwxrwx 1 root root  25523 Jun 29 14:00 catalina.sh
-rwxrwxrwx 1 root root   1664 Jun  2 21:08 catalina-tasks.xml
-rwxrwxrwx 1 root root   2007 Jun 28 03:01 ciphers.sh
-rwxrwxrwx 1 root root  25410 Jun  2 21:08 commons-daemon.jar
-rwxrwxrwx 1 root root 211777 Jun  2 21:08 commons-daemon-native.tar.gz
-rwxrwxrwx 1 root root   1932 Jun 28 03:01 configtest.sh
-rwxrwxrwx 1 root root   9110 Jun 28 03:01 daemon.sh
-rwxrwxrwx 1 root root   1975 Jun 28 03:01 digest.sh
-rwxrwxrwx 1 root root   3392 Jun 28 03:01 makebase.sh
-rwxrwxrwx 1 root root   3718 Jun 28 03:01 setclasspath.sh
-rwxrwxrwx 1 root root   1912 Jun 28 03:01 shutdown.sh
-rwxrwxrwx 1 root root   1914 Jun 28 03:01 startup.sh
-rwxrwxrwx 1 root root  46898 Jun  2 21:08 tomcat-juli.jar
-rwxrwxrwx 1 root root   5550 Jun 28 03:01 tool-wrapper.sh
-rwxrwxrwx 1 root root   1918 Jun 28 03:01 version.sh

I've looked at the catalina.sh script, the part which cause the issue is the following:

  if [ -r "$CATALINA_HOME"/bin/setclasspath.sh ]; then
    . "$CATALINA_HOME"/bin/setclasspath.sh
  else
    echo "Cannot find $CATALINA_HOME/bin/setclasspath.sh"
    echo "This file is needed to run this program"
  fi

The -r inside the condition is borked. I've read it looked if the file exists and is readable, it fill all conditions. I've added elif with -a and -f condition and the do return true, but the rights seems to be the issue despite them being set to 777 or not. I've add a whoami inside the script as well, and it's the root user, so not an issue of ownership.

The startup.sh script has a similar issue, with a -x condition, where it cannot find the catalina.sh...

Answer 1

We just stumbled over this very problem today.

We have an Ubuntu 18.04 server that was upgraded from 16.04. The versions of the docker packages read:

docker-ce/now 5:19.03.1~3-0~ubuntu-xenial amd64
docker-ce-cli/now 5:19.03.1~3-0~ubuntu-xenial amd64
docker-compose/bionic,bionic,now 1.17.1-2 all

Kernel is: 4.15.0-154-generic x86_64

On this machine, running a current version of tomcat:9-jre11 [0] results in the same problem as depicted in your question.

To narrow it down, we just started a bash like this:

docker run -it --rm --entrypoint=/bin/bash tomcat:9-jre11

Now here comes the strange behavior you observed, which is completely unrelated to tomcat:

root@f338debf92f6:/usr/local/tomcat# [[ -r /bin/bash ]]
root@f338debf92f6:/usr/local/tomcat# echo $?
1

On any other machine we tested, the result is as expected, eg:

root@0083a80a9ec2:/usr/local/tomcat# [[ -r /bin/bash ]]
root@0083a80a9ec2:/usr/local/tomcat# echo $?
0

Unfortunately I was not able to reproduce the behavior using a freshly installed Ubuntu 18.04. I even downgraded the kernel version and installed docker from the xenial repo.

Trying to google a solution I found: https://github.com/alpinelinux/docker-alpine/issues/156#issuecomment-912645029

So I tried strace, and here the problem is visible:

On our Ubuntu 18.04:

...
read(255, "#!/bin/bash\n[[ -r /bin/bash ]]\n", 31) = 31
faccessat2(AT_FDCWD, "/bin/bash", R_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
read(255, "", 31)                       = 0
...

And on any other machine I tested:

...
read(255, "#!/bin/bash\n[[ -r /bin/bash ]]\n", 31) = 31
faccessat2(AT_FDCWD, "/bin/bash", R_OK, AT_EACCESS) = -1 ENOSYS (Function not implemented)
faccessat(AT_FDCWD, "/bin/bash", R_OK)  = 0
read(255, "", 31) 
...

Researching the faccessat2 system call shows that it should not return EPERM [1]. I could not quite pinpoint where this behavior is introduced - somewhere between glibc and seccomp, but it all boils down to the runtime being too old for this new syscall.

Here are the solutions we came up with:

1. Upgrade your machine - this might not be feasible, though :)
2. Use a tomcat image based on an older version of Debian/Ubuntu. For us tomcat:9.0.64-jre11-openjdk-slim-bullseye worked fine.
3. Run the container using the --privileged switch. This circumvents the syscall privilege problem, but would be generally a bad idea

References

0. digest sha256:f0c2eb420166a7d609c0031699e0778e11256f280cc2bfb5bfd61cde7ae45c61
1. https://man7.org/linux/man-pages/man2/faccessat.2.html

Answer 2

The Problem is descriped here:

https://github.com/docker-library/tomcat/issues/269

The Base Image (Eclipse Temurin) from the Tomcat Container was updated to Ubuntu LTS 22.04 Jammy based Temurin image.

If you use an old Docker Version and libseccomp on your Host you will run into the Problem with the "-r" Flag in bash.

Our Solution was to use the Tomcat tomcat:9-jdk11-temurin-focal

Answer 3

I had the same problem running a tomcat:9-jdk8 image, running on a debian 10.3 VM that was no more up-to-date.

Upgrading the whole system by

sudo apt-get update
sudo apt upgrade
-> reboot VM

solved the problem. Actual versions now: docker-client: 20.10.17, docker engine: 19.03.9, kernel: 4.19.0-21-amd64

Interestigly: The problem only occured when running the image that was built on this outdated system. The 'same' tomcat image built on our jenkins server started without problems on my local outdated VM.