Merge node_modules to single file to use in hosted express js on parse.com

Question

I'm using Parse.com and hosted cloud code. I have a very simple express.js site, but I wan't to have some simple support for web sockets ( express-ws package). But since Parse doesn't handle require() very well for modules in the node_modules folder I have problems.

I'm looking for a way to merge all dependent modules into a single file and also update all require statements for this to work.

I've looked at some solutions that seemed promising at first, like mcjs , bna , browserify , but I ran into problems with each one.

Have anyone had a similar problem and found a solution?

Answer 1

A good first step would be to run npm dedupe which:

Searches the local package tree and attempts to simplify the overall structure by moving dependencies further up the tree, where they can be more effectively shared by multiple dependent packages.

Next, take note of any "unavoidable conflicts" the dedupping process reports.

These will be versions of modules used in multiple places that cannot be coalesced into a single module (where differing non-compatible versions are referenced in different modules). If possible you want to get rid of these, either by up or downgrading one or the other so they can be shared.

So for example, request requires form-data@0.2.0 while superagent requires form-data@0.1.3 . If possible, you'll need to resolve this conflict by either upgrading superagent to use form-data@0.2.0 or downgrading your instance of request to use form-data@0.1.3 .

Finally, you'll need to package the remaining modules into some form that would allow you to access them from your current script as if they were being loaded from the filesystem.

One way to do this would be to use ender which, while focussed on handling package management for the browser, could easily be used combine multiple modules into a single file for use in a cloud-hosted environment.

I've not yet experimented with ender and so cannot speak to how easy or hard it would be to use for your purpose. I do know that it produces a self-running function that will add functions/features to the default scope (the global object in nodejs.)

One of the functions it overwrites is node's require() function, so if you need to use it in your code, you'll need to preserve a copy.

So assuming you have your application encapsulated into its own module 'Foo' which 'requires' a number of modules in the standard way, first you would need to run the ender build function in your module's root directory, and then include it in something like:

var window={}; // required by ender.

/* 
 * if desired, preserve a copy of node's own require function 
 * note that this is only useful if you expect to require other
 * modules in the "usual" way after ender.js has been evaluated.
 */
var origRequire=require;

/* insert ender.js source (produced by `ender build`) here */

/* invoke ender's version of require to load your own module. */
var Foo=require('Foo');

/* invoke your module's entry point */
Foo.run();

Again, I have not tested this, but believe it should work.

UPDATE

Ah...well it turns out that express-ws requires the compilation of a node extension, which cannot work with ender or for that matter, any module management strategy because node extensions have to be compiled for the local system architecture from source code.

But assuming we could get express-ws loaded as expected, running ender build in a new module directory with express and express-ws as requirements reports:

Robs-MacBook-Pro:foo raisch$ ender build
Installing packages: "ender-core ender-commonjs ."...
ender-core@2.0.0 node_modules/ender-core
ender-commonjs@1.0.8 node_modules/ender-commonjs
Successfully finished installing packages
Assembling build...
Your current build command is: ender build
Your current build size is: 317.9 kB raw, 98.8 kB minified and 30 kB gzipped

Active packages:
├── ender-core@2.0.0 - core client library of Ender
├── ender-commonjs@1.0.8 - commonjs module support for ender
└─┬ foo@1.0.0 - 
  ├─┬ express@4.12.4 - Fast, unopinionated, minimalist web framework
  │ ├─┬ accepts@1.2.7 - Higher-level content negotiation
  │ │ ├─┬ mime-types@2.0.11 - The ultimate javascript content-type utility.
  │ │ │ └── mime-db@1.9.1 - Media Type Database
  │ │ └── negotiator@0.5.3 - HTTP content negotiation
  │ ├── content-disposition@0.5.0 - Create and parse Content-Disposition header
  │ ├── content-type@1.0.1 - Create and parse HTTP Content-Type header
  │ ├── cookie@0.1.2 - cookie parsing and serialization
  │ ├── cookie-signature@1.0.6 - Sign and unsign cookies
  │ ├─┬ debug@2.2.0 - small debugging utility
  │ │ └── ms@0.7.1 - Tiny ms conversion utility
  │ ├── depd@1.0.1 - Deprecate all the things
  │ ├── escape-html@1.0.1 - Escape HTML entities
  │ ├─┬ etag@1.6.0 - Create simple ETags
  │ │ └── crc@3.2.1 - Various CRC JavaScript implementations
  │ ├─┬ finalhandler@0.3.6 - Node.js final http responder
  │ │ ├─┬ debug@2.2.0 - small debugging utility
  │ │ │ └── ms@0.7.1 - Tiny ms conversion utility
  │ │ ├── escape-html@1.0.1 - Escape HTML entities
  │ │ └─┬ on-finished@2.2.1 - Execute a callback when a request closes, finishes, or errors
  │ │   └── ee-first@1.1.0 - return the first event in a set of ee/event pairs
  │ ├── fresh@0.2.4 - HTTP response freshness testing
  │ ├── merge-descriptors@1.0.0 - Merge objects using descriptors
  │ ├── methods@1.1.1 - HTTP methods that node supports
  │ ├─┬ on-finished@2.2.1 - Execute a callback when a request closes, finishes, or errors
  │ │ └── ee-first@1.1.0 - return the first event in a set of ee/event pairs
  │ ├── parseurl@1.3.0 - parse a url with memoization
  │ ├── path-to-regexp@0.1.3 - Express style path to RegExp utility
  │ ├─┬ proxy-addr@1.0.8 - Determine address of proxied request
  │ │ ├── forwarded@0.1.0 - Parse HTTP X-Forwarded-For header
  │ │ └── ipaddr.js@1.0.1 - A library for manipulating IPv4 and IPv6 addresses in JavaScript.
  │ ├── qs@2.4.2 - A querystring parser that supports nesting and arrays, with a depth limit
  │ ├── range-parser@1.0.2 - Range header field string parser
  │ ├─┬ send@0.12.3 - Better streaming static file server with Range and conditional-GET support
  │ │ ├─┬ debug@2.2.0 - small debugging utility
  │ │ │ └── ms@0.7.1 - Tiny ms conversion utility
  │ │ ├── depd@1.0.1 - Deprecate all the things
  │ │ ├── destroy@1.0.3 - destroy a stream if possible
  │ │ ├── escape-html@1.0.1 - Escape HTML entities
  │ │ ├─┬ etag@1.6.0 - Create simple ETags
  │ │ │ └── crc@3.2.1 - Various CRC JavaScript implementations
  │ │ ├── fresh@0.2.4 - HTTP response freshness testing
  │ │ ├── mime@1.3.4 - A comprehensive library for mime-type mapping
  │ │ ├── ms@0.7.1 - Tiny ms conversion utility
  │ │ ├─┬ on-finished@2.2.1 - Execute a callback when a request closes, finishes, or errors
  │ │ │ └── ee-first@1.1.0 - return the first event in a set of ee/event pairs
  │ │ └── range-parser@1.0.2 - Range header field string parser
  │ ├─┬ serve-static@1.9.3 - Serve static files
  │ │ ├── escape-html@1.0.1 - Escape HTML entities
  │ │ ├── parseurl@1.3.0 - parse a url with memoization
  │ │ ├─┬ send@0.12.3 - Better streaming static file server with Range and conditional-GET support
  │ │ │ ├─┬ debug@2.2.0 - small debugging utility
  │ │ │ │ └── ms@0.7.1 - Tiny ms conversion utility
  │ │ │ ├── depd@1.0.1 - Deprecate all the things
  │ │ │ ├── destroy@1.0.3 - destroy a stream if possible
  │ │ │ ├── escape-html@1.0.1 - Escape HTML entities
  │ │ │ ├─┬ etag@1.6.0 - Create simple ETags
  │ │ │ │ └── crc@3.2.1 - Various CRC JavaScript implementations
  │ │ │ ├── fresh@0.2.4 - HTTP response freshness testing
  │ │ │ ├── mime@1.3.4 - A comprehensive library for mime-type mapping
  │ │ │ ├── ms@0.7.1 - Tiny ms conversion utility
  │ │ │ ├─┬ on-finished@2.2.1 - Execute a callback when a request closes, finishes, or errors
  │ │ │ │ └── ee-first@1.1.0 - return the first event in a set of ee/event pairs
  │ │ │ └── range-parser@1.0.2 - Range header field string parser
  │ │ └── utils-merge@1.0.0 - merge() utility function
  │ ├─┬ type-is@1.6.2 - Infer the content-type of a request.
  │ │ ├── media-typer@0.3.0 - Simple RFC 6838 media type parser and formatter
  │ │ └─┬ mime-types@2.0.11 - The ultimate javascript content-type utility.
  │ │   └── mime-db@1.9.1 - Media Type Database
  │ ├── vary@1.0.0 - Manipulate the HTTP Vary header
  │ └── utils-merge@1.0.0 - merge() utility function
  └─┬ express-ws@0.2.6 - WebSocket endpoints for express applications
    ├── url-join@0.0.1 - Join urls and normalize as in path.join.
    └─┬ ws@0.4.32 - simple to use, blazing fast and thoroughly tested websocket client, server and console for node.js, up-to-date against RFC-6455
      ├── commander@2.1.0 - the complete solution for node.js command-line programs
      ├── nan@1.0.0 - Native Abstractions for Node.js: C++ header for Node 0.8->0.12 compatibility
      ├── tinycolor@0.0.1 - a to-the-point color module for node
      └── options@0.0.6 - A very light-weight in-code option parsers for node.js.

which looks pretty good as it processed both of my top-level requirements and added all of their requirements as well, to the nth level.

Wrapping the ender.js file the build process produces in:

var window={},
    util=require('util'),
    origRequire=require;

// insert ender.js here

var express=require('express'); // using ender's require

console.log(util.inspect(express,{depth:null}));

and running it produces an error related to a missing mime-db/db.json file which indicates to me that ender was unable to follow the correct includes for the accepts module's use of mime-types which requires mime-db .

So even if the express-ws issue could be resolved, it looks like you'd need to run through all required modules to see which, like accepts , require special handling.