Jupyter Notebooks 中用于机器学习任务的 CPU 使用率/速度

Question

I've just built a brand new powerful desktop PC in order to speed up my scikit learn computations ( specs here ).

I run my code in a Jupyter Notebook and I noticed that if I run the same computation on my old dying laptop and my super-PC the time difference is often small, although on some very demanding cells in can vary from simple to double between the two computers… But my new PC is suppose to be at least 5 times more powerful than my old laptop!

Demanding code example:

y_train_large = (y_train >= 7)
y_train_odd = (y_train % 2 == 1)
y_multilabel = np.c_[y_train_large, y_train_odd]
knn_clf = KNeighborsClassifier()
knn_clf.fit(X_train, y_multilabel)    
y_train_knn_pred = cross_val_predict(knn_clf, X_train, y_multilabel, cv=3)
f1_score(y_multilabel, y_train_knn_pred, average="macro")

Also, when I check the CPU usage during a classifier training for instance, it's very low on both computers (around 5% on the new one and 15-20% on the old one).

I realise that it may be a big noob question but why is that? I read here that Jupyter Notebooks run on the host machine not mine. How to use my own hardware instead? I probably search the wrong way but I cannot find a lot of informations on that subject. What to search for?

Thanks !

Time report for the code above with the small change of setting n_jobs=4 for cross_val_predict():

Computing time for AMD Ryzen 9 3900x 12 cores, RAM 32Go : 12'45'' approx. average CPU usage 15%

Computing time for Intel i7 4750HQ @ 2.00 GHz, RAM 16Go : 19'50'' approx. average CPU usage 62%

Answer 1

OK, so for this particular piece of code, increasing the n_jobs parameter of cross_val_predict() to n_jobs=4 gives a good improvement but still it's unclear to me:

• How to proceed on other machine learning tasks?
• Are there other parameters to tweak in order to get even better results than this?
• How far can we go with n_jobs, how to evaluate the best n_jobs for a given task, how to know when we go too far and the CPU is at risk?

Any expert on those matters is still welcome to answer :)