如何轻松切换 gcloud / kubectl 凭据

Question

At work we use Kube.netes hosted in GCP. I also have a side project hosted in my personal GCP account using Google App Engine (deploy using gcloud app deploy ).

Often when I try to run a command such as kubectl logs -f service-name , I get an error like "Error from server (Forbidden): pods is forbidden: User "my_personal_email@gmail.com" cannot list resource "pods" in API group "" in the namespace "WORK_NAMESPACE": Required "container.pods.list" permission." and then I have to fight with kubectl for hours trying to get it to work.

Can somebody please break it down for a slow person like me, how gcloud and kubectl work together, and how I can easily switch accounts so I can use gcloud commands for my personal projects and kubectl commands for my work projects? I'm happy to nuke my whole config and start from scratch if that's what it takes. I've found various kubectl and gcloud documentation but it doesn't make much sense or talks in circles.

Edit: this is on Linux.

Answer 1

Had the same problem and doing all of the:

gcloud auth login
gcloud auth list
gcloud config set account
gcloud projects list

didn't help. I knew gcloud switched fine as I was able to list other resources with it directly. But it seems kubectl can't pick those changes up automatically, as kubectl/gcloud integration relies on the pre-generated key, which has a 1h expiration(not sure if it's a default but it's what it is on my machine right now). So, on top of setting right user/project/account with gcloud , you should re-generate the creds:

gcloud container clusters get-credentials <my-cluster> --zone <clusters-zone>

Answer 2

I'm in the same boat as you - apps deployed in GKE for work and personal projects deployed in my personal GCP account.

gcloud stores a list of logged in accounts that you can switch between to communicate with associated projects. Take a look at these commands:

gcloud auth login
gcloud auth list
gcloud config set account
gcloud projects list

To work with a specific project under one of your accounts you need to set that configuration via gcloud config set project PROJECT_ID

kubectl has a list of "contexts" on your local machine in ~/.kube/config . Your current context is the cluster you want to run commands against - similar to the active account/project in gcloud .

Unlike gcloud these are cluster specific and store info on cluster endpoint, default namespaces, the current context, etc. You can have contexts from GCP, AWS, on-prem...anywhere you have a cluster. We have different clusters for dev, qa, and prod (thus different contexts) and switch between them a ton. Take a look at the [kubectx project][1] https://github.com/ahmetb/kubectx for an easier way to switch between contexts and namespaces.

kubectl will use the keys from whatever GCP account you are logged in with against the cluster that is set as your current context. ie, from your error above, if your active account for gcloud is your personal but try to list pods from a cluster at work you will get an error. You either need to set the active account/project for gcloud to your work email or change the kubectl context to a cluster that is hosted in your personal GCP account/project.

Answer 3

对我来说，更新 ~/.kube/config 并将到期日期设置为过去的日期可以修复它

Answer 4

TL;DR

1. Use gcloud config configurations to manage your separate profiles with Google Cloud Platform.

2. Add an explicit configuration argument to the cmd-args of your kubeconfig's user to prevent gcloud from producing an access token for an unrelated profile.

    users: - user: auth-provider: config: cmd-args: config --configuration=work config-helper --format=json

---

Can somebody please break it down for a slow person like me, how gcloud and kubectl work together, and how I can easily switch accounts so I can use gcloud commands for my personal projects and kubectl commands for my work projects?

Sure! By following Google's suggested instructions that lead to running gcloud container clusters get-credentials... when configuring a kube.netes cluster, you will end up with a section of your kubeconfig that contains information on what kubectl should do to acquire an access token when communicating with a cluster that is configured with a given user. That will look something like this:

users:
- name: gke_project-name_cluster-zone_cluster-name
  user:
    auth-provider:
      config:
        access-token: &Redacted
        cmd-args: config config-helper --format=json
        cmd-path: /path/to/google-cloud-sdk/bin/gcloud
        expiry: "2022-12-25T01:02:03Z"
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

Basically, this tells kubectl to run gcloud config config-helper --format=json when it needs a new token, and to parse the access_token using the json-path .credential.access_token in the response from that command. This is the crux in understanding how kubectl communicates with gcloud .

Like you, I use google cloud both personally and at work. The issue is that this user configuration block does not take into account the fact that it shouldn't use the currently active gcloud account when generating a credential. Even if you don't use kube.netes in either one of your two projects, extensions in vscode for example might try to run a kubectl command when you're working on something in a different project. If this were to happen after your current token is expired, gcloud config config-helper might get invoked to generate a token using a personal account.

To prevent this from happening, I suggest using gcloud config configuations . Configurations are global configuration profiles that you can quickly switch between. For example, I have two configurations that look like:

> gcloud config configurations list                                               

NAME      IS_ACTIVE  ACCOUNT             PROJECT            COMPUTE_DEFAULT_ZONE       COMPUTE_DEFAULT_REGION
work      False      zev@work.email      work-project       us-west1-a                 us-west1
personal  True       zev@personal.email  personal-project   northamerica-northeast1-a  northamerica-northeast1

With configurations you can alter your kubeconfig to specify which configuration to always use when creating an access token for a given kube.netes user by altering the kubeconfig user's auth-provider.config.cmd-args to include one of your gcloud configurations. With a value like config --configuration=work config-helper --format=json , whenever kubectl needs a new access token, it will use the account from your work configuration regardless of which account is currently active with the gcloud tool.