Not able to create Prometheus in K8S cluster

Question

I'm trying to install Prometheus on my K8S cluster

when I run command

kubectl get namespaces

I got the following namespace:

default       Active   26h
kube-public   Active   26h
kube-system   Active   26h
monitoring    Active   153m
prod          Active   5h49m

Now I want to create the Prometheus via

helm install stable/prometheus --name prom -f k8s-values.yml

and I got error:

Error: release prom-demo failed: namespaces "default" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "default"

even if I switch to monitoring ns I got the same error,

the k8s-values.yml look like following

rbac:
  create: false
server:
  name: server

  service:
    nodePort: 30002
    type: NodePort

Any idea what could be missing here ?

Answer 1

You are getting this error because you are using RBAC without giving the right permissions.

Give the tiller permissions:
taken from https://github.com/helm/helm/blob/master/docs/rbac.md

Example: Service account with cluster-admin role In rbac-config.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

Note: The cluster-admin role is created by default in a Kubernetes cluster, so you don't have to define it explicitly.

$ kubectl create -f rbac-config.yaml
serviceaccount "tiller" created
clusterrolebinding "tiller" created
$ helm init --service-account tiller

Create a service account for prometheus:
Change the value of rbac.create to true :

rbac:
  create: true
server:
  name: server

  service:
    nodePort: 30002
    type: NodePort

Answer 2

Look at prometheus operator to spin up all monitoring services from prometheus stack. below link is helpful https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus/manifests

all the manifests are listed there. go through those files and deploy whatever you need to monitor in your k8s cluster