Use mitmproxy to translate a form key/value to a body post

Question

I am trying to use mitmproxy to adapt requests from a client (monitoring agent) which I can't change.

The client cannot POST data, eg JSON or SOAP XML data to the request. It can only post form name/value pairs (or header name/values).

I'm trying to work around this limitation by:

• getting the client to post a form pair: "phony_key="
• using mitmproxy to remove the key and just post the data

Running the proxy with a replace expression to just strip the "phony_key=" looked good initially

mitmproxy --replace /~bq/phony_param=/ 

... and produced a good result when I invoked a request from curl, something like

curl -d phony_param='<SOAP-ENV:Envelope></SOAP-ENV:Envelope> https://host:port/path

But when invoked from the real client, the request was rejected by the server.

I think it is because the data payload when sent from the real client is urlencoded. Inspecting the request in mitmproxy using the hex option shows that.

0000000000 25 33 43 53 4f 41 50 2d 45 4e 56 25 33 41 45 6e   %3CSOAP-ENV%3AEn
0000000010 76 65 6c 6f 70 65 2b 78 6d 6c 6e 73 25 33 41 53   velope+xmlns%3AS
0000000020 4f 41 50 2d 45 4e 56 3d 25 32 32 68 74 74 70 25   OAP-ENV=%22http%

Whereas the request made by curl invocation shows the un-encoded XML in the hex view.

Since then, I've tried to make an inline script to do the job

def request(context, flow):
    if flow.request.method == "POST":
        form = flow.request.get_form_urlencoded()
        real_post = form.get("phony_param")[0]
        form.__delitem__("phony_param")
        print real_post
        flow.request.content = real_post
        flow.request.set_form_urlencoded(form)

Above I'm trying to remove the form name/value, add the value to the request content and add back the modified form. I don't think I'm on right lines, or at least it's not working yet.

Answer 1

Thanks to mhils for assistance on this.

The problem with the script was that request.set_form_urlencoded and setting request.content both replace the content of the HTTPRequest . You should set one or the other.

I was mistakenly thinking I should code my proxy filter to preserve any additional form parameters in the incoming request. But on reflection, HTTP POST does not mix different content-types. So discarding the form is correct.

The revised script produces a valid request:

def request(context, flow):
    if flow.request.method == "POST":
        form = flow.request.get_form_urlencoded()
        real_post = form.get_first("phony_param")
        if real_post:
            flow.request.content = real_post