Firefox browser disconnect issue when running tests via testem

Question

I am trying to run my ember tests in parallel using ember exam. However when I run the tests I am frequently getting browser disconnect issue, with following log:

[16:41:10][Step 6/6] not ok 561 Firefox - error
[16:41:10][Step 6/6]     ---
[16:41:10][Step 6/6]         message: >
[16:41:10][Step 6/6]             Error: Browser disconnected
[16:41:10][Step 6/6]             Stderr: 
[16:41:10][Step 6/6]              *** You are running in headless mode.
[16:41:10][Step 6/6]             [Parent 12706, Gecko_IOThread] WARNING: pipe error (45): Connection reset by peer: file /build/firefox-8HKHfQ/firefox-57.0+build4/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353
[16:41:10][Step 6/6]             
[16:41:10][Step 6/6]             ###!!! [Parent][MessageChannel] Error: (msgtype=0x24001F,name=PContent::Msg_PreferenceUpdate) Channel error: cannot send/recv
[16:41:10][Step 6/6]             
[16:41:10][Step 6/6]             
[16:41:10][Step 6/6]             ###!!! [Parent][MessageChannel] Error: (msgtype=0x150083,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv
[16:41:10][Step 6/6]             
[16:41:10][Step 6/6]             
[16:41:10][Step 6/6]             ###!!! [Parent][MessageChannel] Error: (msgtype=0x150083,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv
[16:41:10][Step 6/6]             
[16:41:10][Step 6/6]             
[16:41:10][Step 6/6]             
[16:41:10][Step 6/6]         Log: |
[16:41:10][Step 6/6]             { type: 'error', text: 'Error: Browser disconnected' }
[16:41:10][Step 6/6]             { type: 'error',
[16:41:10][Step 6/6]               text: '*** You are running in headless mode.\n[Parent 12706, Gecko_IOThread] WARNING: pipe error (45): Connection reset by peer: file /build/firefox-8HKHfQ/firefox-57.0+build4/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353\n\n###!!! [Parent][MessageChannel] Error: (msgtype=0x24001F,name=PContent::Msg_PreferenceUpdate) Channel error: cannot send/recv\n\n\n###!!! [Parent][MessageChannel] Error: (msgtype=0x150083,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv\n\n\n###!!! [Parent][MessageChannel] Error: (msgtype=0x150083,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv\n\n' }
[16:41:10][Step 6/6]     ...

I am running firefox version 57, version 59 is better but even there I see this issue quite frequently, chrome however seems to fare well but on occasion it fails too. Initially I thought its because I am running in headless mode, but even in non-headless mode things do not get any better. And my testem.js file is:

let today = new Date();
const testWindowWidth = 1920;
const testWindowHeight = 1080;

/**
 * Exports configuration to run tests via testem.
 * There are issues with running the test in firefox in headless mode.
 * Chrome however works fine with the provided configuration.
 * @see : https://github.com/testem/testem/issues/1117
 * */
module.exports = {
    'framework': 'qunit',
    'test_page': [
        'tests/index.html?hidepassed&nojshint'
    ],
    'disable_watching': true,
    'parallel': -1,//sky is the limit
    'launch_in_ci': [
        'Firefox'
    ],
    'launch_in_dev': [
        'Chrome'
    ],
    'browser_args': {
        'Firefox': [
            `-headless`,//comment this line out for debugging purpose
            `-width ${testWindowWidth}`,
            `-height ${testWindowHeight}`
        ],
        'Chrome': [
            '--headless',//comment this line out for debugging purpose
            '--disable-gpu',
            '--remote-debugging-port=9222',
            '--remote-debugging-address=0.0.0.0',
            '--no-sandbox',
            '--user-data-dir=/tmp',
            `--window-size=${testWindowWidth},${testWindowHeight}`
        ]
    },
    'browser_start_timeout': 2 * 60,
    'report_file': `tests/logs/${today.getFullYear()}-${today.getMonth() + 1}-${today.getDate()}/${today}.txt`,
    'browser_disconnect_timeout': 2 * 60
};

I don't know if the problem is with testem configuration or the browser itself! Your help/suggestions will be much appreciated, thanks.

Answer 1

I was not able to solve the issue but I was able to avoid the browser disconnect issue via ember-exam. The trick was to figure out the no of partitions to split out the tests and run just subset of it in parallel such that each ember run has about 200 tests top. We have fairly large ember application with upwards of 3k tests, so I ended up creating shell script to manage the split.

//run-tests.sh 
const exec = require('child_process').exec;
var split = 100;//total no of partitions to split the tests into
var parallelRuns = 4;//total no of partitions to run in parallel at a time
var runTestModule = function(partitionNo) {
    if (partitionNo > split) {
        return;
    }
    var _partitions = [];
    for (var i = 0; i < parallelRuns; i++) {
        _partitions.push(++partitionNo);
    }
    var partitions = _partitions.map(function(index) {
        return "--partition=" + index;
    }).join(" ");
    var command = "./node_modules/.bin/ember exam --path=./dist --split=" + split + " " + partitions + " --parallel";
    console.log(
        "\n===============================================RUNNING COMMAND===============================================================\n" +
        command +
        "\n=============================================================================================================================\n");

    exec(command, function(e, stdout, stderr) {
        if (e instanceof Error) {
            console.error(e);
            //throw e;
        }
        console.log('stdout ', stdout);
        console.log('stderr ', stderr);
        runTestModule(partitionNo);
    });
};

runTestModule(0);