Why Apache spawns too much processes?

Question

I have a WordPress blog on my LAMP server. I must make 20.000 POST requests to WordPress API in a short period of time to create posts (I make migration from other CMS to WordPress).

For this I run my migration script, which needs to hit https://example.com/wp-json/wp/v2/posts endpoint 20.000 times. After ~1000 requests my server slows down, and the average load goes to 15 . When I check top , this is what I get:

 5827 www-data  20   0  943612  73000  33424 S  4.7  3.6   0:24.62 apache2
 5828 www-data  20   0  943960  79112  38768 S  4.0  3.9   0:16.26 apache2
 5861 www-data  20   0  944280  86568  45968 S  4.0  4.3   0:06.47 apache2
 6047 www-data  20   0  943692  77288  37292 S  3.7  3.8   0:00.46 apache2
 5835 www-data  20   0  942212  85760  47096 S  3.3  4.2   0:05.01 apache2
27086 mysql     20   0 1208472 187212  21124 S  3.0  9.3  18:52.19 mysqld
 5863 www-data  20   0  944452 103896  64936 S  2.7  5.1   0:06.97 apache2
 5826 www-data  20   0  942604 102272  63264 S  2.0  5.1   0:06.66 apache2
 5854 www-data  20   0  944252  84776  46180 S  1.3  4.2   0:06.96 apache2
 5967 www-data  20   0  941856  82388  44128 S  1.3  4.1   0:03.98 apache2
 6046 www-data  20   0  943692  77476  37356 S  1.3  3.8   0:00.51 apache2
 5860 www-data  20   0  942224  84668  46048 S  0.7  4.2   0:06.93 apache2
 6059 www-data  20   0  941644  75192  37164 S  0.7  3.7   0:00.20 apache2

When I stop the migration script, those processes are still running and the high server load stays at the same value (around 15 ). Only thing which helps is sudo apachectl restart . But then I reset apache2, run the migration script again, then after 1000 requests loads go high again.

Answer 1

Apache spawns process to handle requests: assuming you're using MPM, you can tune it following the official documentation.( https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers ) Access logs and error logs might help you find out what's happening.

The problem might lie elsewhere, for example on the database: maybe it's not tuned to handle the load WP is generating to store the information you want to save calling the APIs.

Moreover, instead of tuning apache/mysql/mariadb/postgreSQL you can try lowering the frequence the script calls the api. You can try lowring the

Answer 2

You have so many processes because that is how apache has been told to behave. The links @Inc0 provided will give you some clues how you can cap usage. For a faster recovery time you should set a low MaxSpareServers.

and the average load goes to 15

If your load is higher than the number of CPU threads, then the kernel will start pre-empting processes - that means your throughput will drop. By how much, I can't really say. But if this is a one-off exercise, how much time and effort are you prepared to spend getting it to go faster rather than just getting on with load the data / reduced efficiency?

I run my migration script

Is this multi-threaded / using curl_multi-* functions? I wouldn't expect a single threaded operation to cause so much persistent load. You'd get a lot of clues as to what is going on from mod_status. Regardless, it appears you are not using keepalive in your client (or its deisabled on your server). Enabling this at both ends should have a significant impact on Load / performance.

I make migration from other CMS to WordPress

So this is the XY problem.

Wordpress is very slow and resource heavy. Your biggest win would come from bypassing Wordpress and writing the data directly to the database. Second best would be using the built-in bulk import function in Wordpress (it also exports so you see the data structure). Being Wordpress, there are also several third-party plugins you can use (but please uninstall the plugin when you have finished the import.