Prometheus Remote write to AWS API gateway

Question

Currently I'm trying to configure remote-write with Prometheus to send data to an API gateway in AWS.

I know that prometheus can write directly to many different data sources, like influxdb or other databases, however for specific reasons I won't go into it needs to go to an API gateway endpoint. When I've configured the endpoint to accept binary datatypes like the application/x-protobuf sent in the headers of the write request I still get the following error returned to my prometheus agent:

ts=2022-07-08T18:56:25.014Z caller=dedupe.go:112 component=remote level=error remote_name=69e124 url=https://xxxxxx.execute-api.us-east-1.amazonaws.com/ msg="non-recoverable error while sending metadata" count=291 err="server returned HTTP status 415 Unsupported Media Type: "
ts=2022-07-08T18:56:25.995Z caller=dedupe.go:112 component=remote level=error remote_name=69e124 url=https://xxxxxx.execute-api.us-east-1.amazonaws.com/ msg="non-recoverable error" count=348 exemplarCount=0 err="server returned HTTP status 415 Unsupported Media Type: "

Here is how I configured API gateway: Screen shot of API Gateway console configuration for binary datatypes

Is there something obvious I am missing? Is there a better way I can troubleshoot this? I'm a little lost on how to debug this further.

If it matters, the prometheus agent is inside of an EKS cluster, I can post its configuration if that would be helpful

Answer 1

I was able to confirm that I was writing to an endpoint and I reached out to AWS support for their specific issue. Its not in prometheus its an AWS api gateway and its backend integration configuration problem.

The endpoint I'm hitting looks like the following EX: https://xxxxx.execute-api.us-east-1.amazonaws.com/ and the endpoint does recieve the information. I ended up asking AWS and going a bit further into it. Apparently you need to do the following:

From AWS support:

In addition to adding Binary media type for the API, if the backend is expecting a Binary data load then we need to configure Content Handling set to Convert to Binary. This will explicitly set it to binary even if content type header doesn't match.

So I needed to set that in the lambda, which I have yet to do. What I did instead as a work around was spin up an express based API running in a k8s cluster, and act as my middle transformer for the prometheus request, so the express application takes the protobuf and converts it to JSON before forwarding it on to the API to be processed.