Minikube 上的 RabbitMQ 集群 Kubernetes Operator

Question

I'm trying to set up RabbitMQ on Minikube using the RabbitMQ Cluster Operator :

When I try to attach a persistent volume, I get the following error:

$ kubectl logs -f rabbitmq-rabbitmq-server-0

Configuring logger redirection
20:04:40.081 [warning] Failed to write PID file "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-rabbitmq-server-0.rabbitmq-rabbitmq-headless.default.pid": permission denied
20:04:40.264 [error] Failed to create Ra data directory at '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-rabbitmq-server-0.rabbitmq-rabbitmq-headless.default/quorum/rabbit@rabbitmq-rabbitmq-server-0.rabbitmq-rabbitmq-headless.default', file system operation error: enoent
20:04:40.265 [error] Supervisor ra_sup had child ra_system_sup started with ra_system_sup:start_link() at undefined exit with reason {error,"Ra could not create its data directory. See the log for details."} in context start_error
20:04:40.266 [error] CRASH REPORT Process <0.247.0> with 0 neighbours exited with reason: {error,"Ra could not create its data directory. See the log for details."} in ra_system_sup:init/1 line 43
20:04:40.267 [error] CRASH REPORT Process <0.241.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,ra_system_sup,{error,"Ra could not create its data directory. See the log for details."}}},{ra_app,start,[normal,[]]}} in application_master:init/4 line 138
{"Kernel pid terminated",application_controller,"{application_start_failure,ra,{{shutdown,{failed_to_start_child,ra_system_sup,{error,\"Ra could not create its data directory. See the log for details.\"}}},{ra_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,ra,{{shutdown,{failed_to_start_child,ra_system_sup,{error,"Ra could not create its data directory. See the log for details."}

Crash dump is being written to: erl_crash.dump...

The issue is that RabbitMQ is not able to set up its data files in the data directory /var/lib/rabbitmq/mnesia due to a lack of permission.

My initial guess was that I needed to specify the data directory as a volumeMount, but that doesn't seem to be configurable according to the documentation .

RabbitMQ's troubleshooting documentation on persistence results in a 404 .

I tried to find other resources online with the same problem but none of them were using the RabbitMQ Cluster Operator. I plan on following that route if I'm not able to find a solution but I really would like to solve this issue somehow.

Does anyone have any ideas?

The setup that I have is as follows:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq
spec:
  replicas: 1
  service:
    type: NodePort
  persistence:
    storageClassName: local-storage
    storage: 20Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rabbitmq-pvc
spec:
  storageClassName: local-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: rabbitmq-pv
spec:
  storageClassName: local-storage
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 20Gi
  hostPath:
    path: /mnt/app/rabbitmq
    type: DirectoryOrCreate

To reproduce this issue on minikube:

1. Install the rabbitmq operator:

kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"

2. Apply the manifest file above

kubectl apply -f rabbitmq.yml

3. Running kubectl get po displays a pod named rabbitmq-rabbitmq-server-0 .

4. Running kubectl logs -f rabbitmq-rabbitmq-server-0 to view the logs displays the above error.

Answer 1

As I alread suggested in comments, you can solve it running:

minikube ssh -- sudo chmod g+w /mnt/app/rabbitmq/ 

---

Answering to your question:

Is there a way I can add that to my manifest file rather than having to do it manually?

You can override the rabbitmq statefulset manifest fields to change last line in initContainer command script from chgrp 999 /var/lib/rabbitmq/mnesia/ to this: chown 999:999 /var/lib/rabbitmq/mnesia/ .

Complete RabbitmqCluster yaml manifest looks like following:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq
spec:
  replicas: 1
  service:
    type: NodePort
  persistence:
    storageClassName: local-storage
    storage: 20Gi
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers: []
            initContainers:
            - name: setup-container
              command:
              - sh
              - -c
              - cp /tmp/rabbitmq/rabbitmq.conf /etc/rabbitmq/rabbitmq.conf && chown 999:999
                /etc/rabbitmq/rabbitmq.conf && echo '' >> /etc/rabbitmq/rabbitmq.conf ; cp /tmp/rabbitmq/advanced.config
                /etc/rabbitmq/advanced.config && chown 999:999 /etc/rabbitmq/advanced.config
                ; cp /tmp/rabbitmq/rabbitmq-env.conf /etc/rabbitmq/rabbitmq-env.conf && chown
                999:999 /etc/rabbitmq/rabbitmq-env.conf ; cp /tmp/erlang-cookie-secret/.erlang.cookie
                /var/lib/rabbitmq/.erlang.cookie && chown 999:999 /var/lib/rabbitmq/.erlang.cookie
                && chmod 600 /var/lib/rabbitmq/.erlang.cookie ; cp /tmp/rabbitmq-plugins/enabled_plugins
                /etc/rabbitmq/enabled_plugins && chown 999:999 /etc/rabbitmq/enabled_plugins
                ; chown 999:999 /var/lib/rabbitmq/mnesia/ # <- CHANGED THIS LINE

Answer 2

I had the same issue when deploying RabbitMQ in kubernetes inside Vagrant (not minikube though). I was using this setup. I tried running sudo chmod g+w /mnt/app/rabbitmq/ but had no luck... Eventually gave up and ended up running minikube inside vagrant using this box and everything worked perfectly fine out of the box! Didn't have to do anything special... Not event manually creating the persistent volume...

inside my nodes

Answer 3

我在实时版本中遇到了这个问题，当时 minikube 不允许运行 SSH 命令。所以我所做的是将 chmod 运行到我的主机路径配置程序并重新创建了我的 rabbitmq-cluster

chmod 777 /tmp/hostpath-provisioner/default/*

Answer 4

I found the answer for this issue. It happen when there is few nodes in cluster. Solution is to add securityContext: {}

https://github.com/rabbitmq/rabbitmq-website/blob/3ee8e72a7c4fe52e323ba1039eecbf3a67c554f7/site/kubernetes/operator/using-on-openshift.md#arbitrary-user-ids