Limit/Controlled kubectl exec command

Question

I am still bit of kube.netes newbe. But I am looking for a way to give developers controlled access to kubectl exec command. I want to give them run most of the read-only command but prevent some high risk command and also prevent interactive download/install etc. Also want to log all their action during sessions for audit purpose.

I am not seeing any straight forward way to do it using rbac. Also not seeing those options in rancher either. I am looking for some guidance and direction to achieve such capability.

I am sure some of you have achieved it some way.

Answer 1

Kube.netes RBAC can only validate whenever you can or cannot exec into pods, (by checking create verb on pods/exec resource), after that it switches to SPDY protocol and passes your input and returning back output from analog of docker exec on your container runtime, without actually caring about what's going in and out

With rbac you also have to specify pod name, which might be problematic if you are using Deployments, where each new revision will generate a different pod name. Since pattern matching is not implemented in rbac - you would have to change your role every time new pod name is generated.

So the answer is "No, you can' do it with rbac"

An alternative solution would be to use some kind of CI/CD (jenkins,gitlab-ci etc.) or orchestration tool (rundeck, ansible-tower etc) where you will create some kind of script, where your developers would pass arguments to a job, controlled by you, ie

kubectl exec deploy/foo -- /bin/bar baz "$DEV_ARGUMENT"

Which, essentially, means, that you would be responsible for managing access to that job/script, creating and maintaining serviceAccount for that script, etc.

If you are afraid of image mutability, ie you don't want your developers to install something in running container, but otherwise are okay with giving them shell on it (remember, they can still read any secrets/env vars/configMaps and even serviceAccount tokens that pod uses of you mount them by default), you should consider the following:

1. Don't run your containers as root . Try to use images, that support rootles operation, and then either specify correct non-root UID in runAsUser field in securityContext , or configure runAsNonRoot: true flag to deny containers running as root.
2. Better general solution would be to utilize PodSecurityPolicy (deprecated, removed in 1.25), Pod Security Admission or some 3rd party admission contoller like OPA Gatekeeper to deny containers running as root in your namespace
3. You can also make your pods immutable by using readOnlyRootFilesystem in security context, which will deny write operation to pod ephemeral storage (but if you mounted any volume as RW - they still will be accessible to write operations). Feasibility of this approach depends on whenever your apps use some kind of temporary files of not

Relevant links:

• kube.netes RBAC role verbs to exec to pod
• https://github.com/kube.netes/kube.netes/issues/44703#issuecomment-324826356 - issue, discussing current rbac limitations
• https://itnext.io/how-it-works-kubectl-exec-e31325daa910
• https://erkanerol.github.io/post/how-kubectl-exec-works/ - bot links explaining how exec actually works