Kubernetes cluster best practice

Question

I am working on a new project with Kubernetes and I need three environments: DEV,QA and PROD.

What is most recommended, create Multiple Clusters or create one big cluster separating environments by namespace.

Answer 1

Are you just going to have a single prod cluster or multiple prod clusters? One thing to consider is that updating the cluster management software (to a new k8s release) can impact your application. If you only plan to have a single prod cluster, I'd recommend running qa and dev separately so that you can upgrade those clusters first to shake out any issues. If you are going to have multiple prod clusters, then you can upgrade them one at a time to ensure application availability and sharing the clusters between environments makes a lot more sense.

Answer 2

Namespaces will not bring you isolation, at the moment it's just a different subdomain in dns. It's better to have namespace per application. I highly recommend you to have two clusters for prod (in case of updating k8s) and one-two for dev/qa.

Answer 3

Take a look at this blog post: Checklist: pros and cons of using multiple Kubernetes clusters, and how to distribute workloads between them .

I'd like to highlight some of the pros/cons:

Reasons to have multiple clusters

• Separation of production/development/test: especially for testing a new version of Kubernetes, of a service mesh, of other cluster software
• Compliance: according to some regulations some applications must run in separate clusters/separate VPNs
• Better isolation for security
• Cloud/on-prem: to split the load between on-premise services

Reasons to have a single cluster

• Reduce setup, maintenance and administration overhead
• Improve utilization
• Cost reduction

Considering a not too expensive environment, with average maintenance, and yet still ensuring security isolation for production applications, I would recommend:

• 1 cluster for DEV/QA (separated by namespaces, maybe even isolated, using Network Policies, like in Calico )
• 1 cluster for PROD

Answer 4

Definitely concur that you want multiple clusters: anything critical to k8s that may fail during an upgrade or because you screw up somewhere will affect the whole cluster.

for example, I had an issue with DNS which wrecked havoc in my cluster; all namespaces were affected.

Upgrades are usually not a big deal but one day you might hit a roadblock; if kubelet fails for too long your pods will get killed.

So it's best to upgrade your test/dev environments and iron things out there before upgrading in prod.