Persistent Storage for Kubernetes in Production

Question

I experimenting and learning Rancher/Kubernetes. In a very short eBook I read this.

"In a Kubernetes Cluster, it can be desirable to have persistent storage available for applications to use. As we do not have a Kubernetes Cloud Provider enabled in this cluster, we will be deploying the nfs-server-provisioner which will run an NFS server inside of our Kubernetes cluster for persistent storage.

This is not a production-ready solution by any means, but helps to illustrate the persistent storage constructs. "

I configured nfs-server-provisioner in Rancher and everything works as expected. But here the question.

For my "production" homelab, I prepared 5x Bare Metal Server and installed Rancher on top of a Kubernates, also I created an RKE2 Cluster with

• etcd Node
• control plane Node
• worker Node 1
• worker Node 2

I dont use AWS, Azure or any Cloud Solutions.

What would be a "production ready solution" for my stack. And why exactly is "nfs-server-provisioner" not a "production ready solution"?

Answer 1

Without seeing the full text this is only a guess, but based on that quote only using nfs-server-provider isn't providing "true" and reliable persistence.

nfs-server-provider launches the NFS server within the cluster, which means also its data is within the kubernetes' storage system. There's no real persistence there: instead the persistence, availability and security of NFS based persistent volumes depend on how the nfs-server-provider stores the data. You definitely lose production-readiness if served NFS data is stored by the provider in a way that is not highly available - say, on a local hostpath on each node. Then again if nfs-server-provider is using a reliable storage class, why not cut the overhead and use that storage class directly for all persistent volumes? This might be what your quoted text refers to.

I'd also like to note (at least at the side), that using NFS as a storage class when the NFS server resides inside the same cluster might potentially mean asking for trouble. If provisioning of nfs-server-provider on nodes fails for some reason (even a trivial one, like not being able to fetch the image) you'd lose access to NFS based persistent volumes, sending all pods relying on NFS volumes on crashloops. This is however true of Longhorn, OpenEBS and other cluster-residing storage classes, too.

Making a production ready setup would require you to at least configure the nfs-server-provider itself to use a production-grade storage backend or use a highly available external NFS.

Also note that for production grade you should have at least two control plane and three etcd nodes instead of just one and one (never use an even number of etcd nodes.), One node can run multiple nodes, so with your equipment I'd probably go for two nodes running both control plane and etcd. two "pure" worker nodes and one node doing all three, The last isn't exactly recommended. but in a homelab environment would give you more workers when testing with pod replicas.