Spring boot services deploying using Docker-compose - memory issue: Resource exhaustion event

Question

I am building an application using spring-boot microservices and deploying into the docker container. I am using docker-compose mechanism to deploy the services into the containers.

• My docker setup is on Windows-10 Home Edition.
• WSL2 setting is also enabled.

I am using:

• AdoptOpenJDK 11.0.10+9
• Spring boot 2.4.2
• Docker desktop Engine 20.10.5
• Compose 1.29.0

When I am going to deploy the applications into the container using docker-compose up command, I have below error in one of the service while deployment.

Resource exhaustion event: the JVM was unable to allocate memory from the heap.

And prints some data.

I have modified the docker settings of WSL2 and allocate more memory to it from 4GB to 8GB but still it failing.

While analysis of logs I found one thing:

The services which are deployed are not taking the memory as mentioned in the docker-compose file. Here are the logs and file:

Logs:

springboot-naming-server_1       | Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx86677K -XX:MaxMetaspaceSize=118122K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 700M, Thread Count: 250, Loaded Class Count: 18441, Headroom: 0%)
springboot-api-gateway_1         | Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx290293K -XX:MaxMetaspaceSize=119306K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 700M, Thread Count: 50, Loaded Class Count: 18650, Headroom: 0%)
role-service_1                   | Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx69418K -XX:MaxMetaspaceSize=135381K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 700M, Thread Count: 250, Loaded Class Count: 21488, Headroom: 0%)
user-service_1                   | Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx48733K -XX:MaxMetaspaceSize=156066K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 700M, Thread Count: 250, Loaded Class Count: 25140, Headroom: 0%)

The -Xmx value different. The container memory is already allocated.

Docker-compose file:

version: '3.8'

services:
  rabbitmq:
    image: rabbitmq:3.8.14-management
    mem_limit: 300m
    ports:
      - "5672:5672"
      - "15672:15672"
    networks:
      - test-network

  zipkin-server:
    image: openzipkin/zipkin:2.23
    ports:
      - "9411:9411"
    deploy:
      resources:
        limits:
          cpus: 0.15
          memory: 300m
    networks:
      - test-network
    depends_on:
      - rabbitmq
    restart: unless-stopped
    environment:
      - RABBIT_URI=amqp://guest:guest@rabbitmq:5672
    
  redis-cache-server:
    image: redis
    command: ['redis-server', '--requirepass', 'aaaxxxcccvvv']
    deploy:
      resources:
        limits:
          memory: 356m
    ports:
      - "6379:6379"
    networks:
      - test-network

  springboot-naming-server:
    image: myapp/test-app-naming-server:1.0
    deploy:
      resources:
        limits:
          cpus: 0.15
          memory: 700m
    ports:
      - "8761:8761"
    networks:
      - test-network
      
  springboot-api-gateway:
    image: myapp/test-app-api-gateway:1.0
    deploy:
      resources:
        limits:
          cpus: 0.15
          memory: 700m
    ports:
      - "8765:8765"
    networks:
      - test-network
    depends_on: 
      - springboot-naming-server
      - rabbitmq
    environment:
      - EUREKA.CLIENT.SERVICEURL.DEFAULTZONE=http://springboot-naming-server:8761/eureka
      - RABBIT_URI=amqp://guest:guest@rabbitmq:5672
      - SPRING_RABBITMQ_HOST=rabbitmq
      - SPRING_ZIPKIN_SENDER_TYPE=rabbit

  role-service:
    image: myapp/test-roleservice:1.0
    deploy:
      resources:
        limits:
          memory: 700m
    ports:
      - "8083:8083"
    networks:
      - test-network
    depends_on: 
      - springboot-naming-server
      - springboot-api-gateway
      - rabbitmq
    environment:
      - EUREKA.CLIENT.SERVICEURL.DEFAULTZONE=http://springboot-naming-server:8761/eureka
      - SPRING.ZIPKIN.BASE-URL=http://zipkin-server:9411
      - RABBIT_URI=amqp://guest:guest@rabbitmq:5672
      - SPRING_RABBITMQ_HOST=rabbitmq
      - SPRING_ZIPKIN_SENDER_TYPE=rabbit
            
  user-service:
    image: myapp/test-userservice:1.0
    deploy:
      resources:
        limits:
          memory: 700m
    ports:
      - "8084:8084"
    networks:
      - test-network
    depends_on: 
      - springboot-naming-server
      - springboot-api-gateway
      - redis-cache-server
      - rabbitmq   
    environment:
      - EUREKA.CLIENT.SERVICEURL.DEFAULTZONE=http://springboot-naming-server:8761/eureka
      - SPRING.ZIPKIN.BASE-URL=http://zipkin-server:9411
      - RABBIT_URI=amqp://guest:guest@rabbitmq:5672
      - SPRING_RABBITMQ_HOST=rabbitmq
      - SPRING_ZIPKIN_SENDER_TYPE=rabbit

  web-app:
    build:
      context: .
      dockerfile: Dockerfile.Web
    ports: 
      - "8081:8080"
    networks:
      - test-network
    depends_on: 
      - springboot-naming-server
      - springboot-api-gateway
      - redis-cache-server
      - rabbitmq      
    command: ['catalina.sh', 'run']
    environment:
      - EUREKA.CLIENT.SERVICEURL.DEFAULTZONE=http://springboot-naming-server:8761/eureka
      - SPRING.REDIS.HOST=redis-cache-server
      - SPRING.ZIPKIN.BASE-URL=http://zipkin-server:9411
      - RABBIT_URI=amqp://guest:guest@rabbitmq:5672
      - SPRING_RABBITMQ_HOST=rabbitmq
      - SPRING_ZIPKIN_SENDER_TYPE=rabbit
    
networks:
  test-network:

I am not able to understand why each service wont get the memory equally. And because of that my last service "userservice" always failed with the error related to momory(already mentioned above).

Can someone give me some pointer where I need to check the things like any configuration or setting related to docker or spring-boot etc.

Thanks,

Atul

Answer 1

I am able to solve it.

The problem is like I mentioned in my question. While at the time of startup each service takes lot of memory (see the logs in the question section), so there is no memory remaining for rest of the services and because of that rest of the service failed while starting with the message related to memory.

The changes I made to the docker-compse.yml file and those are:

environment:
      - JAVA_TOOL_OPTIONS=-Xmx128000K
deploy:
  resources:
    limits:
      memory: 800m

These 2 changes I made and then the memory allocation is allocated as per the requirement and every service now starts.

What I found is(might be wrong observation) the docker compose start the service it look for that service only and allocate memory. And when other services look for the memory they failed to get memory as earlier service almost took lot of memory.

Thanks,

Atul