Cypress result shows skipped tests as pending

Cypress results (with run command) shows skipped tests as pending

Current behavior:

(Results)
  ┌──────────────────────────────────────────────────┐
  │ Tests:        4                                  │
  │ Passing:      3                                  │
  │ Failing:      0                                  │
  │ Pending:      1                                  │
  │ Skipped:      0                                  │
  │ Screenshots:  0                                  │
  │ Video:        true                               │
  │ Duration:     29 seconds                         │
  │ Spec Ran:     xxx\xxx\Page2b.spec.js │
  └──────────────────────────────────────────────────┘

Result file: (see skipped="1" in Mocha Tests)

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="29.914" tests="4" failures="0" skipped="1">
  <testsuite name="Root Suite" timestamp="2019-01-08T18:14:13" tests="0" failures="0" time="0">
  </testsuite>
  <testsuite name="Page2b" timestamp="2019-01-08T18:14:13" tests="1" failures="0" time="8.691">
    <testcase name="Page2b should see entered data from previous page" time="8.691" classname="should see entered data from previous page">
    </testcase>
  </testsuite>
  <testsuite name="Valid data" timestamp="2019-01-08T18:14:22" tests="2" failures="0" time="21.223">
    <testcase name="Page2b Valid data goes to next page after entering valid data" time="7.554" classname="goes to next page after entering valid data">
    </testcase>
    <testcase name="Page2b Valid data should see previously entered data when I go back from next page" time="13.669" classname="should see previously entered data when I go back from next page">
    </testcase>
  </testsuite>
  <testsuite name="Invalid data" timestamp="2019-01-08T18:14:43" tests="1" failures="0" time="0">
  </testsuite>
</testsuites>

Desired behavior:

Cypress results should show 0 pending and 1 skipped tests

Steps to reproduce: (app code and test code)

cypress.json:

    "reporter": "junit",
    "reporterOptions": {
        "mochaFile": "results/output.[hash].xml"
    },

test:

describe('Page2b', function () {
    beforeEach(() => {
        cy.fixture('correctAnswers.json').as('answers')
        cy.login()
    })

    context('Valid data', function() {
        it('goes to next page after entering valid data', function () {
            // terefere
        })

        it('should see previously entered data when I go back from next page', function () {
            // terefere
        })
    })

    context('Invalid data', function () {
        it.skip('shows required validation texts after clicking on Next link on empty form', function () {
            // terefere
        })
    })

    it('should see entered data from previous page', function () {
        // terefere
    })
})

Note

This same issue when running:

context.skip('Invalid data', function () {
        it('shows required validation texts after clicking on Next link on empty form', function () {
            // terefere
        })
    })

Versions

Cypress 3.1.4, Chrome 71, Windows 10

Issue Analytics

State:
Created 5 years ago
Reactions:47
Comments:27 (6 by maintainers)

Top GitHub Comments

60reactions

morficuscommented, Feb 28, 2019

+1 for this being confusing.

To the untrained eye, the term “pending” implies that the test is still running or has not been ran yet (as in it is queued to run in the future). Meanwhile the term “skipped” would imply that it was not ran and will not be ran. This is especially confusing because the mocha syntax is it.skip.

My initial thoughts would be to:

option 1 rename “skipped” to “excluded” and “pending” to “skipped”

option 2 consider the things currently called “skipped” to be failed tests … since the reason they did not run would be due to a failing hook. then rename “pending” to “skipped” and keep “pending” for things with no callbacks (or get ride of it all together)

35reactions

brian-manncommented, Jul 31, 2019

I’ve written an extensive answer in our internal chat detailing the history as to how we arrived at the nomenclature we use. It’s long, and complex, and we’re in an awkward position because of what mocha chose to do - it’s not something we can fix on our end unless we deviate from mocha completely - which would end up creating its own big can of worms - or introduce new unique test states (see below).

The problem is that we inherit all of the mocha reporters and we use the spec reporter by default. These mocha reporters are coupled to the definition of what a pending test is which mocha determines. Mocha decided to use the it.skip API design, but internally refers to them as pending tests in the reporter.

Deviating from this nomenclature would create a conflict from the default spec reporter, or other custom reporters you may use. You’d see the mocha stats summary indicating numbers that don’t match up to our own summary at the end.

Because we don’t control mocha reporters - there’s not really much we can do about this. We’d either have to deviate from mocha’s own API’s - such as rewriting it.skip, or we’d have to rewrite and control every single mocha reporter and 3rd party mocha reporter (which is not going to happen).

In addition, Mocha has no concept of accurately depicting truly “skipped” tests. That is the one thing that we ourselves do - when a hook fails we skip the remaining tests in that suite. Sure - I agree we could probably rename this state to another word that would perhaps alleviate some confusion. But no matter what - we inherit Mocha’s it.skip and pending state problems.

One idea we’ve batted around would be to expand the number of test states to help describe and more accurately model the actual runtime behavior of each test.

A more comprehensive list of test states that could be proposed:

Passed (a test that ran to completion including all its applicable hooks)
Failed (a test that failed in a hook or its own test body)
Pending (a test that had no implementation - i.e. no callback function)
Skipped (a test that was programmatically skipped via it.skip, or this.skip(), or our own programmatic API’s)
Bypassed / Excluded / Aborted / Not Ran (a test that was not run because a hook failed in the current suite thus “bypassing/excluding/aborting” the remaining tests in the suite)

This would enable us to separate out the reason a test didn’t run based on the cause.

Unfortunately, the upgrade path here would be pretty harsh, as the current definitions look like this:

Passed (a test that ran to completion including all its applicable hooks)
Failed (a test that failed in a hook or its own test body)
Pending (a test that had no implementation - i.e. no callback function, a test that had it.skip or this.skip())
Skipped (a test that was not run because a hook failed in the current suite thus “bypassing/excluding/aborting” the remaining tests in the suite)

Top Results From Across the Web

Cypress Test Statuses | Better world by better software

After the Cypress spec completes every test has one of the 4 statuses: passing, failing, pending, or skipped. Let's look into each status....

Skip Pending and Skipped tests from HTML report in Cypress

As I have integration with mochawesome and allure both shows Pending tests in corresponding HTML report as well. enter image description here.

Writing and Organizing Tests

After the Cypress spec completes every test has one of 4 statuses: passed, failed, pending, or skipped. Passed. Passed tests have successfully completed...

Dashboard Introduction - cypress

Dashboard Introduction · See the number of passing, failing, pending or skipped tests · Obtain the entire stack trace of the failed tests...

Cypress Tests | Testkube Documentation - GitHub Pages

To create a new Cypress test, you will need a Git repository with an example Cypress project. ... Spec Tests Passing Failing Pending...