谷歌数据融合 CPU

Question

I have a problem when I trying to deploy a downstream pipeline, the error from logs is this:

PROVISION task failed in REQUESTING_CREATE state for program run program_run:default.ListaNomi1_v3.-SNAPSHOT.workflow.DataPipelineWorkflow.182bbf2c-576b-11ec-8095-da8d4f8ab0b3 due to Dataproc operation failure: INVALID_ARGUMENT: Multiple validation errors: - Insufficient 'CPUS' quota. Requested 10.0, available 3.0. - Insufficient 'CPUS_ALL_REGIONS' quota. Requested 10.0, available 7.0. - Insufficient 'IN_USE_ADDRESSES' quota. Requested 3.0, available 1.0. - This request exceeds CPU quota. Some things to try: request fewer workers (a minimum of 2 is required), use smaller master and/or worker machine types (such as n1-standard-2)..

I'm trying to change the worker and Master nodes configuration but it always Fail, I can't modify the quota beacause I m not the leader and he says that can't change.

Answer 1

To process data with Cloud Data Fusion you need a cluster. Two options are:

• Ephemeral cluster when it's created for each pipeline run. This is the one you are trying to use, but it needs compute quotas to create a cluster
• Static cluster (Existing Dataproc). In this case the cluster is created beforehand and you simply "points" your Pipeline to use it by creating and selection provisioning profile. This can be an option to prevent quota issues during pipeline start. But such a static cluster would incur costs while it's running, even without any jobs.