How to emit custom metrics from SpringBoot application and use it in PCF Autoscaler

Question

I am following this example for emitting the custom metrics and followed these details for registering the metrics in PCF.

Here is the code:

@RestController
public class CustomMetricsController {

    @Autowired
    private MeterRegistry registry;     

    @GetMapping("/high_latency")
    public ResponseEntity<Integer> highLatency() throws InterruptedException {
        int queueLength=0;
        Random random = new Random();
        int number = random.nextInt(50);
        System.out.println("Generate number is : "+number);
        if(number % 2 == 0) {
            queueLength=99;
        } else {
            queueLength=200;
        }
        return new ResponseEntity<>(queueLength, null, HttpStatus.OK);
    }
}

application.yml:

management:
  endpoints:
    web:
      exposure:
        include: "metrics,prometheus"
  endpoint:
    metrics:
      enabled: true
    prometheus:
      enabled: true

SecurityConfiguration class:

build.gradle dependencies part:

Steps I followed to register the custom metrics after app deployment to PCF:

1. Installed metric-registrar on my local machine
2. Registered endpoint that emitting some Integer(which I am using for Autoscaler Rule) cf register-metrics-endpoint api high_latency After this step I can see one Custom-User-Provided service is bounded with my app in PCF.
3. Installed log-cache plugin on my local to verify the metrics endpoint and here are the logs
4. At last I added the rule in Autoscaler for custom metric.

This is the error I am getting in Autoscaler event history.

EVENT HISTORY Most Recent: Wed July 17, 2019 at 11:20 AM Autoscaler did not receive any metrics for high_latency during the scaling window. Scaling down will be deferred until these metrics are available.

Answer 1

I looked into this a big. A few things here...

1. I believe your registration command is wrong, at least for the sample app you referenced.

   You're using cf register-metrics-endpoint api high_latency , which means you have an app named api and an endpoint on that app at high_latency which exports metrics using the Prometheus format. For this sample app, the path should be /actuator/prometheus , according to the README & my brief test.

   When I used the command cf register-metrics-endpoint app-name /actuator/prometheus , I was able to see the custom metrics in the output from cf tail & I do not see them in your included output.

   Ex: (not showing up in your screenshot)

    2019-07-17T22:54:37.83-0400 [custom-metrics-demo/0] GAUGE tomcat_global_request_max_seconds:2.006000 2019-07-17T22:54:37.83-0400 [custom-metrics-demo/0] GAUGE system_cpu_count:4.000000 2019-07-17T22:54:37.83-0400 [custom-metrics-demo/0] COUNTER tomcat_sessions_created_sessions_total:12 2019-07-17T22:54:37.83-0400 [custom-metrics-demo/0] COUNTER custom_metric_total:15 2019-07-17T22:54:37.83-0400 [custom-metrics-demo/0] GAUGE jvm_gc_pause_seconds_sum:0.138000 2019-07-17T22:54:37.83-0400 [custom-metrics-demo/0] COUNTER jvm_gc_pause_seconds_count:25 

2. In the referenced sample app, there is no metric called high_latency , so that won't work as the name of your metric, as it will never get generated by the app. You can see all the metrics if you access the /actuator/prometheus endpoint in your browser. The Prometheus format is text based and pretty easy to read.

3. The last one is tricky and not documented but I can see it in the code for Autoscaler (at the time of writing this). When Autoscaler polls for metrics from LogCache, it only pulls GAUGE & TIMER events, not COUNTER events. If you were to try and use custom_metric from the demo app that would not work because it is a COUNTER metric. You'll just see the message about Autoscaler not seeing any metric events during the scaling window. Make sure that you are picking a GAUGE metric to use for scaling.

4. It also does not appear that Autoscaler supports using metrics with associated tags. If for example, you wanted to use tomcat_servlet_request_seconds_sum{name="dispatcherServlet",} , I don't think there's a way to tell Autoscaler that the name tag is required to be a certain value. That info is in LogCache, but I don't think Autoscaler uses it at this time. My quick glance through the code, at the time I write this, seems to indicate it's only looking at the metric name. If you're creating a custom metric, this won't matter. Just don't use any tags for the metric.

Hope that helps!