kubeflow 部署 gcp 端点 controller 失败

Question

I am trying to deploy kubeflow on gcp using official guidelines https://www.kubeflow.org/docs/distributions/gke/deploy/deploy-cli/

I tried three times but it seems there is a problem with endpoints controller. When checked by: kubectl -n kubeflow get all

All pods are running except the

    NAME                                                             READY   STATUS             RESTARTS   AGE
pod/admission-webhook-deployment-667bd68d94                1/1     Running            
pod/cache-deployer-deployment-75ccdc98b4                   2/2     Running            
pod/cache-server-56f78bf64b                                2/2     Running            
pod/centraldashboard-5fdbd9b744                            1/1     Running            
pod/cloud-endpoints-controller-5f7dbc6fc8                  0/1     ImagePullBackOff

Pod desciption says that it failed to resolve reference "gcr.io/cloud-solutions-group/cloud-endpoints-controller:0.2.1": unexpected status code [manifests 0.2.1]: 403 Forbidden

I am new to kubeflow but despite retrying this three times it always results in the same issue.

Answer 1

You can clone the repo and build the image yourself and push it to your container registry. This is one workaround to fix this until the official image is back.

git clone https://github.com/jlewi/cloud-endpoints-controller.git
cd cloud-endpoints-controller
git checkout 0.2.1
docker build . -t <YOUR DOCKER REGISTRY>/cloud-endpoints-controller:0.2.1
docker push <YOUR DOCKER REGISTRY>/cloud-endpoints-controller:0.2.1

And this use the new image in your pod spec.

Answer 2

Thank you for posting this issue! I have posted a mitigation solution here in https://github.com/kubeflow/gcp-blueprints/issues/343#issuecomment-1028488756 . I am planning to fix this issue in the coming release.