Gunicorn / flask API暴露sklearn模型不起作用

Question

I can't seem to figure this out. I've got a model trained with scikit-learn, saved to a .pkl file, and I want to make an API that would make predictions based on it.

I already have the code that makes predictions and it runs fine from console/unit-tests. To speed up predictions I'm splitting the data (thousands of image patches) and spreading the load using joblib / multiprocessing .

I'm setting JOBLIB_START_METHOD=forkserver since scikit-learn hangs if used from within a multiprocessing process.

I've got an API done with flask which uses this code, and when run with flask's dev server it works just fine. Now I'm trying to host the flask app within gunicorn and it's not working at all.

If I use the default workers, then it just hangs with no errors when trying to predict, much like if I hadn't set the 'forkserver' multiprocessing. I'm running gunicorn like this:

JOBLIB_START_METHOD=forkserver gunicorn -w 2 -b 0.0.0.0:$PORT --timeout 3600 web.app:app

I also tried using the gevent backend. This actually does work but it's very slow, and it prints this:

Multiprocessing backed parallel loops cannot be nested below threads, setting n_jobs=1

So, any ideas on getting this to work in a way that there's multiple web workers running (I don't think that's the case with flask's dev server) and with a request being able to leverage joblib / multiprocessing ? thanks

Answer 1

Gevent won't work with joblib since it spawns thread(s) to handle requests concurrently (Refer this discussion ) and that's what your warning actually says. Secondly, it's very slow because joblib converts your parallel calls into sequential calls and executes them (Refer to this discussion ).

I did the face the same problem while performing parallelism using joblib. Although I didn't use sklearn, I think the following command should work for you as well:

gunicorn -b 0.0.0.0:$SERVICE_PORT --workers=2 -t $SERVICE_TIMEOUT rest_api:app

If you want to have a look at the complete source code, you can follow it here .