为什么node.js没有使用所有可用带宽？

Question

I have this sample code in coffescript for node.js . What I want to do is to parallel download a lot of objects from S3.

I have a list of users that contains a list of friends , and I want to download all these friends as fast as possible. The sample code waits 1 second to simulate how much it spends downloading an user object , and it downloads an object from S3 to simulate the download of a friend object .

async = require 'async'
aws = require 'aws-sdk'
cluster = require 'cluster'
config = require './config'

USERS   = 30
FRIENDS = 300

class TestSpeed

  constructor: () ->
    @s3 = new aws.S3 accessKeyId:config.S3_KEY, secretAccessKey:config.S3_SECRET

  start: (next) =>
    async.map([0...USERS], @downloadUser, next)

  downloadUser: (x, next) =>
    console.log "Starting to download user #{x}"
    setTimeout(
      =>
        console.log "User downloaded"
        @downloadFriends(next)
      , 1000
    )

  downloadFriends: (next) =>
    console.log "Starting to download friends"
    async.map([0...FRIENDS], @downloadFriend, next)

  downloadFriend: (x, next) =>
    console.log "Starting to download friend #{x}"
    @s3.getObject Bucket:config.BUCKET, Key:config.UID, (err, data) ->
      return console.log err if err? 
      console.log "Friend downloaded"
      next()

if cluster.isMaster
  console.log("starting at master process...")
  cluster.fork() for [0...4]

console.log "init"
new TestSpeed().start (err, result) =>
  return console.trace err if err?
  console.log "OK"

What I expected to happen is node to use all the bandwidth available as I'm creating 4 processes in a 4 core machine. But what I get is a download of around 50Mbps, that doubles to 100Mbps with 8 processes (more than the cores available!).

I thought what node.js uses all the resources available, but not the CPU nor the network are working at 100%. What am I missing?

Answer 1

I guess you're limited by maxSockets.

Try require('http').globalAgent.maxSockets = 1000 at the beginning of your file.

Look at this question Optimizing Node.js for a large number of outbound HTTP requests?