在Kubernetes上自动化部署

Question

I am interested in know what is the best way to automate YAML deployments in Kubernetes

I have a cert-manager helm installation and a kong YAML manifest and an Ingress resource created, installed and working in a Kubernetes cluster

This deployment provides me TLS encryption in order to get the https protocol to my application service which also is installed via helm (I've created a helm chart for it)

My objective is to find the way of executing those YAML files that I have created, and some helm commands of an automatic way and not execute manually the steps and the process.

Maybe is important to keep in mind that all these cert-manager, kong and Ingress deployment is associated to my helm chart service application which I've created, so that I have been deploying the following:

1. I've created a helm chart of my service application and I've installed it via helm install ... command

2. I've installed kong and kong-ingress-controller from a YAML manifest using Postgres like external service.

3. I've installed cert-manager via helm adding its helm repo add jetstack https://charts.jetstack.io and executing

helm install \
    --name cert-manager \
    --namespace cert-manager \
    --version v0.7.0 \
    jetstack/cert-manager

4. And also, I have created an Ingress and KongIngress resources to allow the access to my helm chart application service and allow to kong manage the ingress operations and other actions like the routes, certificates and service creation, and also associate somethings like basic-auth plugin to my service application

All this process was made via kubectl apply ... executing YAML files and also helm .... command from a CLI.

This mean, that currently this process is highly dependent of an human events, and I would like to automatize somethings like the kong and cert-manager installation process and even if is possible the Ingress creation.

Do I am right when I think that this may be possible?

What is the best way to do it?

I have been reading and looking in some places, and some people also have orient me in the following alternatives:

• From my helm chart application ?

  I have been reading that helm has something named hooks

The hooks itself describe the following:

A kind of mechanism to allow chart developers to intervene at certain points in a release's life cycle. For example, you can use hooks to:

Load a ConfigMap or Secret during install before any other charts are loaded.

Execute a Job to back up a database before installing a new chart, and then execute a second job after the upgrade in order to restore data.

Run a Job before deleting a release to gracefully take a service out of rotation before removing it

Could be the helm hooks approach a good option? I think that with them I can automatize or at least that the helm chart application take over the secrets creation and some pre/post-deployment .

Can I consider the kong , cert-manager and Ingress resources operations (commands and YAML files execution) like operations or actions to be managed by helm hooks?

• Terraform approach: Providing the deployment (some steps or completely) from it.

There is a Helm and Kubernetes providers available from terraform. I could explore that possibility of install these software packages from terraform scripts.

This terraform approach could be useful to install my helm chart application and the cert-manager helm

Kong also like terraform provider

Currently there are some third party repositories to work with kong from terraform. Terraform Provider Kong it's more proper to me due to I am using Azure Kubernetes service, and the different resource configurations which are possible to implement looks very good. I am not sure if I would have to inject manually somethings like certificates , route resources when I work with the kong-ingress-controller and cert-manager communication, but even this terraform kong provider allow me the possibility of import a existing routes terraform import kong_route.<route_identifier> <route_id>

Could be apply it this to certificates and other resources?

• Using python scripts

What about of automate deployments via python? How are going the things with this alternative? Is possible do it interacting from python to kubernetes and executing the commands operations which I have described above ( kubectl and helm to create resources from YAML files and helm charts available in local and remote repositories)

In affirmative case ... How to can I address this option?

I have found this option which use ansible but it involve some other things like Automate broker ansible and Kubernetes service catalog, which are unknown for me at the moment.

• Using Jupyter Notebooks.

Can I create those resources ( Ingress and kong and cert-manager installation and configuration) via jupyter notebooks, writing down directly to Kubernetes cluster? Is this possible? How to can I do it?

So, I think that the terraform approach (using the helm and kong providers would be very useful tools from a infrastructure as a code perspective, but I am not sure if I would be choose it, maybe could there is less complex alternatives?

Somebody has performed automation of resources and things in Kubernetes cluster. I suppose that it is a normal or expected process to do, although is my first time.

If someone can point me in the right approach to solve my particular scenario I will be highly grateful. :)

Answer 1

Unfortunately you cannot control the order of installation of dependent Helm charts. Helm groups together the chart's templates and all of its dependencies' templates and applies them as if they came from the same chart (see here for more details on the ordering).

You could write a Python script that installs the charts one after the other using PyHelm (full disclosure - I'm one of the maintainers of the package).

In addition, you can use Helm hooks in order to control the order of the installation - that can take care of your additional yamls (eg using the post-install/post-upgrade hooks).

Answer 2

A possible solution is to use initContainer that will wait for other services to be UP.

Typically, for a given A and B application, you will use helm or event better helmfile to deploy them both at the same time. But for the pod of B, you will add an initContainer, that will check say every 10 seconds if service A is started. By doing this, you are sure that service B will only start when service A is fully started and ready. You can have a look at https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ . This requires to configure livenessProbe and readynessProbe.

Note that this will not really order chart installation, simply ensure that each service will wait for dependendancies before start. And you can install many chart at the same time using helmfile ( https://github.com/roboll/helmfile ).