`npm uninstall` hangs (or very slow) without apparent activity

Question

I've always found npm uninstall to take a surprisingly long time and am trying to troubleshoot. I started an uninstall of four packages ~25 minutes ago, and it appears stalled with no progress, no significant CPU activity, and no apparent disk activity (using iotop ). I'm at a loss for what the issue could be.

Here's the current output (anonymized) with a loglevel of 'silly':

$ npm uninstall --save-dev -ddd gulp-autoprefixer gulp-sass gulp-sequence gulp-sourcemaps
npm info it worked if it ends with ok
npm verb cli [ '/path/to/home/bin/node-v0.12.2-linux-x64/bin/node',
npm verb cli   '/path/to/home/bin/node-v0.12.2-linux-x64/bin/npm',
npm verb cli   'uninstall',
npm verb cli   '--save-dev',
npm verb cli   '-ddd',
npm verb cli   'gulp-autoprefixer',
npm verb cli   'gulp-sass',
npm verb cli   'gulp-sequence',
npm verb cli   'gulp-sourcemaps' ]
npm info using npm@2.7.4
npm info using node@v0.12.2
npm verb unbuild node_modules/gulp-autoprefixer
npm verb unbuild node_modules/gulp-sass
npm verb unbuild node_modules/gulp-sequence
npm verb unbuild node_modules/gulp-sourcemaps
npm sill gentlyRm /path/to/home/myproject/node_modules/gulp-autoprefixer is being purged from base /path/to/home/myproject
npm verb gentlyRm don't care about contents; nuking /path/to/home/myproject/node_modules/gulp-autoprefixer
npm sill gentlyRm /path/to/home/myproject/node_modules/gulp-sass is being purged from base /path/to/home/myproject
npm verb gentlyRm don't care about contents; nuking /path/to/home/myproject/node_modules/gulp-sass
npm sill gentlyRm /path/to/home/myproject/node_modules/gulp-sequence is being purged from base /path/to/home/myproject
npm verb gentlyRm don't care about contents; nuking /path/to/home/myproject/node_modules/gulp-sequence
npm sill gentlyRm /path/to/home/myproject/node_modules/gulp-sourcemaps is being purged from base /path/to/home/myproject
npm verb gentlyRm don't care about contents; nuking /path/to/home/myproject/node_modules/gulp-sourcemaps
npm sill vacuum-fs purging /path/to/home/myproject/node_modules/gulp-autoprefixer
npm sill vacuum-fs purging /path/to/home/myproject/node_modules/gulp-sass
npm sill vacuum-fs purging /path/to/home/myproject/node_modules/gulp-sequence
npm sill vacuum-fs purging /path/to/home/myproject/node_modules/gulp-sourcemaps

...it has been stalled there since shortly after I started it. Earlier attempts to remove packages one at a time worked, but still stalled at vacuum-fs step for some time (order of ~30 seconds). I'm running Ubuntu 14.04 LTS on a quad-core Intel i5 with 8 GB of memory, Node v0.12.2, and npm 2.7.4.

Does anyone know what the problem could be or how to troubleshoot further?

(As an aside, I'm not sure if this is the appropriate StackExchange site for tooling questions, so please recommend otherwise if not!)

---

Edit: I did some sleuthing using strace per @dekkard's answer below; here's an excerpt from the area where the slowdown happens (using strace -f -t -o out.log npm uninstall -D gobble as an example):

...
18327 17:00:52 rmdir("/path/to/home/myproject/node_modules/gobble/node_modules/minimatch" <unfinished ...>
18318 17:00:52 read(8, "\1\0\0\0\0\0\0\0", 1024) = 8
18318 17:00:52 epoll_wait(5,  <unfinished ...>
18325 17:00:55 <... rmdir resumed> )    = -1 ENOTEMPTY (Directory not empty)
18325 17:00:55 write(8, "\1\0\0\0\0\0\0\0", 8) = 8
18325 17:00:55 rmdir("/path/to/home/myproject/node_modules/gobble/node_modules/mkdirp" <unfinished ...>
18318 17:00:55 <... epoll_wait resumed> {{EPOLLIN, {u32=8, u64=8}}}, 1024, -1) = 1
18318 17:00:55 read(8, "\1\0\0\0\0\0\0\0", 1024) = 8
18318 17:00:55 epoll_wait(5,  <unfinished ...>
18328 17:00:58 <... rmdir resumed> )    = -1 ENOTEMPTY (Directory not empty)
18328 17:00:58 write(8, "\1\0\0\0\0\0\0\0", 8) = 8
18318 17:00:58 <... epoll_wait resumed> {{EPOLLIN, {u32=8, u64=8}}}, 1024, -1) = 1
18328 17:00:58 rmdir("/path/to/home/myproject/node_modules/gobble/node_modules/promise-map-series" <unfinished ...>
18318 17:00:58 read(8, "\1\0\0\0\0\0\0\0", 1024) = 8
18318 17:00:58 epoll_wait(5,  <unfinished ...>
18326 17:01:01 <... rmdir resumed> )    = -1 ENOTEMPTY (Directory not empty)
18326 17:01:01 write(8, "\1\0\0\0\0\0\0\0", 8) = 8
18318 17:01:01 <... epoll_wait resumed> {{EPOLLIN, {u32=8, u64=8}}}, 1024, -1) = 1
...

When viewed live in the console, there stall of ~3 seconds between each epoll_wait(5, and any subsequent output. Thoughts?

Answer 1

For future reference, the root issue was a combination of the following two things:

1. The node rimraf module (used by npm and numerous other packages) operates by first attempting to remove a directory, and only reading its contents if rmdir errors with ENOTEMPTY (see this line in the rimraf source ).

2. My project folder is on an AFS volume, and by default it seems AFS fileservers throttle clients that issue too many failing RPCs in a row (hat-tip to the folks at MIT's Student Information Processing Board for pointing that out via email). Specifically, the defaults institute a 3-second delay for more than 10 failing RPCs in a row.

The net result is that rimraf and packages that rely on it slow to a crawl on AFS volumes, at least on my machine... while it seems there are OS optimizations to make at least some commands error early before hitting the fileserver (eg http://milek.blogspot.com/2014/01/mkdir-performance.html ) that doesn't seem to be happening for rmdir on my system based on the logs above.

I've opened a rimraf issue here: https://github.com/isaacs/rimraf/issues/73

Answer 2

I'd suggest finding the PID of your npm uninstall and attach to it with strace or ltrace :

strace -c -p <PID>

http://linux.die.net/man/1/strace

-c
Count time, calls, and errors for each system call and report a summary on program exit

Also, you can filter what calls to trace (ie file, network, signal, etc.)

UPDATE:

18318 17:00:52 epoll_wait(5, <unfinished ...>

You can try to scroll back and find the call to epoll_create() that returned epollfd == 5 which is later passed to epoll_wait() . Checking its surrounding calls might provide some clues.