Custom Class-Style commands returning a promise reject does not stop execution

[humphreyn]

Description of the bug/issue

When I call my custom command that returns a promise.reject
Then I expected the scenario.result.status to equal "FAILED" in the cucumber AfterStep
And I expect any following cucumber steps to be skipped
But instead the AfterStep scenario.result.status equals "PASSED"
And any following cucumber steps are executed

Steps to reproduce

Any custom command that returns a promise.reject causes this error
The attached script uses a custom command that always returns a promise reject
We have 1.x custom commands that wait for a condition and return a promise.reject if that condition is not met within the timeout period
Version 2.x does not seem to handle custom commands promise.rejects at all

Sample test

//NightwatchJS Custom command
module.exports = class MyCustomCommand {
	command(callback) {
		if (typeof callback === "function") {
			callback.call(this.api);
		}
		const result = {
			"status": -1,
			"value": false,
			"error": new Error("Promise Class Command 'myCustomCommand' returns promise.reject")
		}
		return Promise.reject(result);
	}
};


//Cucumber step definition file
//cucumber.steps.js
const {Given, Then, When, AfterStep} = require('@cucumber/cucumber');

AfterStep(function (scenario) {
  if (['I call my custom command'].includes(scenario.pickleStep.text)) {
    console.log(`Expected the "scenario.result.status" to equal "FAILED" but got "${scenario.result.status}"`)
  }
})

Given(/^I have navigated to "([^"]*)"$/, function(url) {
  return browser.navigateTo(url);
});

When(/^I call my custom command$/, function() {
  return browser.myCustomCommand();
});

Then(/^I expected this step to be skipped$/, function() {
  return browser.assert.fail("I expect this step to be skipped");
});


//Cucumber Feature file
//customCommand.feature
Feature: Custom Command Promise reject

  Scenario: Test custom command promise rejection
    Given I have navigated to "https://nightwatchjs.org"
    When I call my custom command
    And I expected this step to be skipped

Command to run

npx nightwatch --verbose

Verbose Output

Launching up to 1 concurrent test worker processes...

 Running:  default: cucumber.steps.js 

DevTools listening on ws://127.0.0.1:65317/devtools/browser/596c685a-4458-46b5-98b4-41f94dfe380f


×  default: cucumber.steps.js    
 - Starting ChromeDriver on port 9515... 
 Starting ChromeDriver with server_path=C:\temp\test\node_modules\chromedriver\lib\chromedriver\chromedriver.exe... 
   Request POST /session   
 { 
      capabilities: { 
        firstMatch: [ {} ], 
        alwaysMatch: { browserName: 'chrome', 'goog:chromeOptions': {} } 
      } 
   } 
   Response 200 POST /session (999ms)
 {
      value: {
        capabilities: {
          acceptInsecureCerts: false,
          browserName: 'chrome',
          browserVersion: '113.0.5672.127',
          chrome: {
            chromedriverVersion: '113.0.5672.63 (0e1a4471d5ae5bf128b1bd8f4d627c8cbd55f70c-refs/branch-heads/5672@{#912})',
            userDataDir: 'C:\\Users\\HUMPHR~1\\AppData\\Local\\Temp\\scoped_dir25592_1990293323'
          },
          'goog:chromeOptions': { debuggerAddress: 'localhost:65317' },
          networkConnectionEnabled: false,
          pageLoadStrategy: 'normal',
          platformName: 'windows',
          proxy: {},
          setWindowRect: true,
          strictFileInteractability: false,
          timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },
          unhandledPromptBehavior: 'dismiss and notify',
          'webauthn:extension:credBlob': true,
          'webauthn:extension:largeBlob': true,
          'webauthn:extension:minPinLength': true,
          'webauthn:extension:prf': true,
          'webauthn:virtualAuthenticators': true
        },
        sessionId: 'caab856e953d2fdc13753f3207c6ad82'
      }
   }
 i Connected to ChromeDriver on port 9515 (1083ms).
 Using: chrome (113.0.5672.127) on WINDOWS.
 Received session with ID: caab856e953d2fdc13753f3207c6ad82 
 
 .
  
  → Running command: navigateTo ('https://nightwatchjs.org')
   Request POST /session/caab856e953d2fdc13753f3207c6ad82/url  
 { url: 'https://nightwatchjs.org' }
   Response 200 POST /session/caab856e953d2fdc13753f3207c6ad82/url (2945ms)
 { value: null }
  → Completed command: navigateTo ('https://nightwatchjs.org') (2951ms)
 .
  
  → Running command: myCustomCommand ()
  Error
    Promise Class Command 'myCustomCommand' returns promise.reject
  → Completed command: myCustomCommand () (1ms)
 Expected the "scenario.result.status" to equal "FAILED" but got "PASSED"
 .
 { value: null }
  → Completed command: quit () (167ms)
 .

 Failures:
 1) Scenario: Test custom command promise rejection # features\customCommand.feature:3
    √ Before # node_modules\nightwatch\cucumber-js\_setup_cucumber_runner.js:6
    √ Given I have navigated to "https://nightwatchjs.org" # step_definitions\cucumber.steps.js:9
    √ When I call my custom command # step_definitions\cucumber.steps.js:13
    × And I expected this step to be skipped # step_definitions\cucumber.steps.js:17
        I expect this step to be skipped
    √ After # node_modules\nightwatch\cucumber-js\_setup_cucumber_runner.js:67
 1 scenario (1 failed)
 3 steps (1 failed, 2 passed)
 0m05.350s (executing steps: 0m05.332s)

 Wrote HTML report file to: C:\temp\test\tests_output\nightwatch-html-report\index.html

 Wrote Rerun Json report file to: C:\temp\test\tests_output\minimal_report.json

Nightwatch Configuration

module.exports = {
	"src_folders": ["step_definitions"],
	"page_objects_path": ["page-objects"],
	"custom_commands_path": ["commands"],
	"custom_assertions_path": [],
	"plugins": [],
	"globals_path": "",
	"webdriver": {},
	"test_workers": {
		"enabled": true
	},
	"test_settings": {
		"default": {
			"disable_error_log": false,
			"launch_url": "http://google.co.uk",
			"report_command_errors": false,
			"end_session_on_fail": true,
			"skip_testcases_on_fail": true,
			"screenshots": {
				"enabled": false,
				"path": "screens",
				"on_failure": true
			},
			"desiredCapabilities": {
				"browserName": "chrome"
			},
			"webdriver": {
				"start_process": true,
				"server_path": ""
			},
			"test_runner": {
				"type": "cucumber",
				"options": {
					"feature_path": "features"
				}
			}
		},
		"chrome": {
			"desiredCapabilities": {
				"browserName": "chrome",
				"goog:chromeOptions": {
					"w3c": true,
					"args": []
				}
			},
			"webdriver": {
				"start_process": true,
				"server_path": "",
				"cli_args": []
			}
		}
	}
};

Nightwatch.js Version

2.6.21

Node Version

16.18.0

Browser

chrome 113.0.5672.127

Operating System

No response

Additional Information

No response

[AutomatedTester]

This could be a duplicate of #2510

[humphreyn]

Hi @AutomatedTester I am not sure. Existing API commands do stop the tests from continuing, but when you create a custom command that follows the documentation it does not.

[humphreyn]

@swrdfish , is there any timeline as to when this will be fixed as we cannot migrate from version 1.* until this is resolved

[swrdfish]

@humphreyn this is a complex bug, we are looking into it but cannot promise on a timeline yet.

[beatfactor]

@humphreyn Is it happening with the default test format or is it only when using the Cucumber runner?

[humphreyn]

@beatfactor , I will set up a test can get back to you as we currently only use the Cucumber runner.

[memihai]

@beatfactor, using the NW runner doesn't stop execution. To the default NW project installed with "npm init nightwatch" I added the below as follows:

//nightwatch/page-objects/google/search.js

const searchCommands = {
  dismissModal(){
    this.waitForElementVisible('@button', 5000)
        .click('@button');
    this.api.myCustomCommand();
    return this; // for command-chaining
  },
  submit() {
    this.waitForElementVisible('@submitButton', 1000)
      .click('@submitButton');
    
    this.pause(1000);

    return this; // for command-chaining
  }
};


module.exports = {
  url: 'https://google.no',

  commands: [
    searchCommands
  ],

  elements: {
    searchBar: {
      selector: 'textarea#APjFqb'
    },

    submitButton: {
      selector: 'input.gNO89b'
    },
    button: {
      selector:'button#W0wltc'
    }
  }
};

//nightwatch/examples/with-page-objects/google.js

describe('google search with consent form - page objects', function() {
  const homePage = browser.page.google.search(); // first page-object

  before(async () => {
    homePage.navigate();
  });


  after(async (browser) => browser.quit());

  it('should find nightwatch.js in results', function(browser) {
    homePage.dismissModal();
    homePage.setValue('@searchBar', 'Nightwatch.js');
    homePage.submit();

    const resultsPage = browser.page.google.searchResults(); // second page-object
    resultsPage.expect.element('@results').to.be.present;

    resultsPage.expect.element('@results').text.to.contain('Nightwatch.js');

    resultsPage.expect.section('@menu').to.be.visible;

    const menuSection = resultsPage.section.menu;
    menuSection.expect.element('@all').to.be.visible;
  });
});

//nightwatch/custom-commands/myCustomCommand.js

module.exports = class MyCustomCommand {
  command(callback) {
    if (typeof callback === "function") {
      callback.call(this.api);
    }
    const result = {
      "status": -1,
      "value": false,
      "error": new Error("Promise Class Command 'myCustomCommand' returns promise.reject")
    }
    return Promise.reject(result);
  }
};

//output after running: npx nightwatch ./nightwatch/examples/with-page-objects/google.js

google search with consent form - page objects] Test Suite
───────────────────────────────────────────────────────────────────────────────
ℹ Connected to ChromeDriver on port 9515 (2418ms).
  Using: chrome (113.0.5672.126) on MAC OS X.

  ℹ Loaded url https://google.no in 1219ms

  Running should find nightwatch.js in results:
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ✔ Element <button#W0wltc> was visible after 50 milliseconds.
  Error
   Promise Class Command 'myCustomCommand' returns promise.reject
  ✔ Element <input.gNO89b> was visible after 131 milliseconds.
  ✔ Expected element @results <#rso> to be present (66ms)
  ✔ Expected element @results <#rso> text to contain: "Nightwatch.js" (138ms)
  ✔ Expected element @menu <#hdtb-msb> to be visible (43ms)
  ✔ Expected element <Section [name=menu],Element [name=@all[0]]> to be visible (152ms)

  FAILED: 1 errors and  6 passed (2.199s)

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  ️TEST FAILURE (7.331s): 
   - 1 error during execution; 
   - 0 assertions failed; 6 passed

   ✖ 1) with-page-objects/google

   – should find nightwatch.js in results (2.199s)


  - OTHER ERRORS:
  Error
    Error
   Promise Class Command 'myCustomCommand' returns promise.reject

FYI @humphreyn

[memihai]

@humphreyn If you are migrating a framework you might also be interested in this issue. This one is my blocker and you might hit it. I am also migrating a ~2000 test framework from 1.*.

While working on the migration I had also hit the issue you had logged here so thank you for logging it.