Nginx upstream retry based on response

Question

Not all API operations are retriable, so backend send a header retriable = true if the operations can be retried.

How can I say Nginx: retry if the upstream response contains the retriable header?

upstream mash {
    ip_hash;
    server 127.0.0.1:8081;
    server 192.168.0.11:8081;
}

server {

    location / {
        if ($request_method = POST ) {
          proxy_next_upstream error;
        }

        if ($request_method = PUT ) {
          proxy_next_upstream error;
        }

        proxy_pass http://mash/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;        
    }
}

The logic I want to implement is the following:

if(response.hasHeader('Retriable') {
   do_retry
} else{
   return_backend_response;
}

Answer 1

The Retrier Service

Create a second upstream retrier with a very simple web server running that forwards all requests back to nginx on a special port. Then nginx will forward the requests to the mash upstream. When the response comes back to nginx, the response will get sent to retrier and the logic for retrying can be executed there.