VertexAI Pipelines: The provided location ID doesn't match the endpoint

Question

I have successfully run the following pipeline that creates a dataset, trains a model, and deploys to an endpoint using VertexAIs pipeline tool when everything is based in us-central1. Now, when I change the region to europe-west2, I get the following error:

debug_error_string = "{"created":"@1647430410.324290053","description":"Error received from peer
ipv4:172.217.169.74:443","file":"src/core/lib/surface/call.cc","file_line":1066,
"grpc_message":"List of found errors:\t1.Field: name; Message: 
The provided location ID doesn't match the endpoint. The valid location ID is `us-central1`.\t","grpc_status":3}"

This error occurs after the dataset is created in europe-west2, and before the model starts to train. Here is my code:

#import libraries
from typing import NamedTuple
import kfp
from kfp import dsl
from kfp.v2 import compiler
from kfp.v2.dsl import (Artifact, Input, InputPath, Model, Output, 
                        OutputPath, ClassificationMetrics, Metrics, component)
from kfp.v2.components.types.artifact_types import Dataset
from kfp.v2.google.client import AIPlatformClient
from google.cloud import aiplatform
from google_cloud_pipeline_components import aiplatform as gcc_aip
from google.api_core.exceptions import NotFound

@kfp.dsl.pipeline(name=f"lookalike-model-training-v2",
                  pipeline_root=PIPELINE_ROOT)
def pipeline(
    bq_source: str = f"bq://{PROJECT_ID}.{DATASET_ID}.{TABLE_NAME}",
    display_name: str = DISPLAY_NAME,
    project: str = PROJECT_ID,
    gcp_region: str = "europe-west2",
    api_endpoint: str = "europe-west2-aiplatform.googleapis.com",
    thresholds_dict_str: str = '{"auPrc": 0.5}',
):
            
    dataset_create_op = gcc_aip.TabularDatasetCreateOp(
        project=project,
        display_name=display_name, 
        bq_source=bq_source,
        location = gcp_region
    )

    training_op = gcc_aip.AutoMLTabularTrainingJobRunOp(
        project=project,
        display_name=display_name,
        optimization_prediction_type="classification",
        budget_milli_node_hours=1000,
        location=gcp_region,
        predefined_split_column_name="set",
        column_transformations=[
            {"categorical": {"column_name": "agentId"}},
            {"categorical": {"column_name": "postcode"}},
            {"categorical": {"column_name": "isMobile"}},
            {"categorical": {"column_name": "gender"}},
            {"categorical": {"column_name": "timeOfDay"}},
            {"categorical": {"column_name": "set"}},
            {"categorical": {"column_name": "sale"}},
        ],
        dataset=dataset_create_op.outputs["dataset"],
        target_column="sale",
    )

compiler.Compiler().compile(
    pipeline_func=pipeline, package_path="tab_classif_pipeline.json"
)

ml_pipeline_job = aiplatform.PipelineJob(
    display_name=f"{MODEL_PREFIX}_training",
    template_path="tab_classif_pipeline.json",
    pipeline_root=PIPELINE_ROOT,
    parameter_values={"project": PROJECT_ID, "display_name": DISPLAY_NAME},
    enable_caching=True,
    location="europe-west2"
)
ml_pipeline_job.submit()

As previously mentioned, the dataset gets created so I suspect that the issue must lie in training_op = gcc_aip.AutoMLTabularTrainingJobRunOp

I tried providing another endpoint: eu-aiplatform.googleapis.com which yielded the following error:

google.api_core.exceptions.InvalidArgument: 400 List of found errors: 1.Field: name; Message: 
The provided location ID doesn't match the endpoint. The valid location ID is `us-central1`.

Fail to send metric: [rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create 
denied (or the resource may not exist).; rpc error: code = PermissionDenied desc = Permission monitoring.timeSeries.create
 denied (or the resource may not exist).]

I understand that I am not passing api-endpoint to any of the methods above, but I thought I'd highlight that the error changed slightly.

Does anyone know what the issue may be? Or how I can run gcc_aip.AutoMLTabularTrainingJobRunOp in europe-west2 (or EU in general)?

Thanks

Answer 1

Try Updating the pipeline component using the command:

pip3 install --force-reinstall google_cloud_pipeline_components==0.1.3