How to properly forward requests through proxies with MITMProxy?

Question

Trying to use MITMProxy to do custom forwarding to requests made from the Firefox browser, so that they go through one of several proxies selected at runtime. It is performing too slow for our purposes. Please bear in mind we are running this in Python 2.7.

The process is as follows:

1. Firefox sends request to configured MITMProxy.
2. MITMProxy takes the request from Firefox and generates a requests request and gets the data from the target server through a given proxy (which is not controlled by us and require authentication).
3. The response from the proxy-forwarded request gets converted into a response for the browser.
4. MITMProxy returns the data to the browser.

The situation seems to be that this process is too slow, which I believe could be for a number of reasons. It could be that there are settings enabled which bring down performance (such as too much logging, for example), the procedure being used is not the right one for the job (totally plausible) or something completely different.

How can we make this run faster?

Thanks very much! Any and all suggestions will be appreciated!

Answer 1

In this particular case, we were using the script feature of MITMProxy, which meant every modified request was executed synchronously (ie, we could not use proper asynchronous behavior). This naturally became an issue once we started using the scripts with more clients.

As @Puciek mentioned in his comment, this was more a design issue than a problem with the library.