How to Stop Suite Execution after First Failure in TestNG?

A TestNG class can have different tests like test1, test2, test3 etc. There could be some failure while running the test suite and user may get failures in between of @Test methods. Once a test method gets failed, it skip remaining code of the @Test method and moves to the next @Test method. However, user may want to skip remaining all @Test methods after getting 1st failure.

There are 2 most popular solution for such use cases:

• Write dependsOnMethods annotation - But this solution works only if user knows exact dependent method/s otherwise in large suite it would be very clumsy.

• Using IInvokedMethodListener to add a common logic across suite.

In this article, let's analyse how to stop site execution after 1st failure using IInvokedMethodListener.

Approach/Algorithm to solve this problem

• Step 1: import org.testng.annotations.Test for TestNG.

• Step 2: Write an annotation as @test in NewTest class

• Step 3: Create a method for the @Test annotation as test1.

• Step 4: Repeat the steps for test2 and test3. Add a failure assertion at test2.

• Step 5: Now, create a ListenerClass that uses IInvokedMethodListener to check the status of previous @Test method. If it gets failed status, ignore remaining suite.

• Step 6: Now create the testNG.xml and add Listener class.

• Step 7: Now, run the testNG.xml or directly testNG class in IDE or compile and run it using command line.

Example

The following code to create a TestNG class and displays the Listener functionality:

src/NewTest.java

import org.testng.annotations.Test;
public class NewTest {
    @Test()
    public void testCase1() {
        System.out.println("in test case 1 of NewTest");
    }
    @Test()
    public void testCase2() {
        System.out.println("in test case 2 of NewTest");
	  assert false;
    }
    @Test()
    public void testCase3() {
        System.out.println("in test case 3 of NewTest");

    }
}

src/ListenerClass.java

import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
import org.testng.SkipException;

public class ListenerClass implements IInvokedMethodListener {

    private boolean hasFailures = false;

    @Override
    public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
        synchronized (this) {
            if (hasFailures) {
                throw new SkipException("Skipping this test");
            }
        }
    }

    @Override
    public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
        if (method.isTestMethod() && !testResult.isSuccess()) {
            synchronized (this) {
                hasFailures = true;
            }
        }
    }
}

testng.xml

This is a configuration file that is used to organize and run the TestNG test cases.

It is very handy when limited tests are needed to execute rather than full suite.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Parent_Suite">
    <listeners>
        <listener class-name="ListenerClass"/>
    </listeners>
    <test name="group test">
        <classes>
            <class name="NewTest" />
        </classes>
    </test>
</suite>

Output

[INFO] Running TestSuite
in test case 1 of NewTest
in test case 2 of NewTest
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 0.671 s <<< FAILURE! - in TestSuite
[ERROR] NewTest.testCase2  Time elapsed: 0.009 s  <<< FAILURE!
java.lang.AssertionError
	at NewTest.testCase2(newTest.java:14)
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   NewTest.testCase2:14
[INFO] 
[ERROR] Tests run: 3, Failures