how do I track the cost of a vertex ai custom job on google cloud platform

Question

I am creating a number of custom jobs on vertex AI using some code very similar to this . I am trying to understand if there is a way to track the cost of each job. I know I can go to the pricing page and try to calculate the costs from there but so far all my attempts to manually calculate them don't add up.

I tried to set up labels in the custom job spec as per here , something very similar is explained in this stackoverflow question. However those labels appear only in the VertexAI->Training->custom jobs page and nowhere else. Can someone please explain me how I can track the costs in the billing account page by using labels or even better if I can make some api call to get the cost generated by a job? Thanks.

Answer 1

I think I found the issue here. "Cost table" and "Billing export" in billing are here to the rescue. The problem is that I didn't have the permissions to see these pages for some reason.

Answer 2

In Vertex AI Pipelines you can use labels. Each pipeline gets vertex-ai-pipelines-run-billing-id label and you can your custom labels. I attach my test example with additional label costcenter . Then you can group your Billing report by label.

The code to implement custom label is very simple.

    import google.cloud.aiplatform as aip
    
    job = aip.PipelineJob(
        display_name="nothing-pipeline-VPC",
        template_path="do_nothing_pipeline.json",
        pipeline_root=pipeline_root_path,
        parameter_values={},
        enable_caching=False,
        location=region,
        labels={"costcenter": "gcpa"}
    )