TestNG - Handle Failed Tests

TenstNg is a robust framework. But at times test cases might fail due to various reasons. Every time tests fail in a suite, TestNG creates a file called testng-failed.xml in the output directory. This XML file contains the necessary information to rerun only these methods that failed, allowing you to quickly reproduce the failures without having to run the entirety of your tests.

TestNG privdes a way to handle this situation. It allows to automatically retry a test whenever it fails. You can use a retry analyzer. When you bind a retry analyzer to a test, TestNG automatically invokes the retry analyzer to determine if TestNG can retry a test case again. Here is how you use a retry analyzer:

• Build an implementation of the interface org.testng.IRetryAnalyzer

• Bind this implementation to the @Test annotation for e.g., @Test(retryAnalyzer = LocalRetry.class)

A org.testng.IRetryAnalyzer interface is as follows:

  public interface IRetryAnalyzer {

    /**
     * Returns true if the test method has to be retried, false otherwise.
     *
     * @param result The result of the test method that just ran.
     * @return true if the test method has to be retried, false otherwise.
     */
    public boolean retry(ITestResult result);
  }

It has only one method.

Create a Class

Let's see the usage of retryAnalyzer in below example. Create a java class TestRetry.java in /work/testng/src. This implements interface IRetryAnalyzer. The retry method make sure that a failed test is retried 3 times. This is because we have specified maxRetryCount = 3;

  import org.testng.IRetryAnalyzer;
  import org.testng.ITestResult;

  public class TestRetry implements IRetryAnalyzer{
  	private int retryCount = 0;
  	  private static final int maxRetryCount = 3;

  	  @Override
  	  public boolean retry(ITestResult result) {
  	    if (retryCount 
Create Test Class
Create a java test class, say, TestClassSample.java in /work/testng/src. We have specified retryAnalyzer value in the @Test annotation
  import org.testng.Assert;
  import org.testng.annotations.Test;

  public class TestClassSample {
  	 @Test(retryAnalyzer = TestRetry.class)
  	  public void test2() {
  	    Assert.fail();
  	  }

  }

Create testng.xml
Next, let's create testng.xml file in /work/testng/src, to execute test case(s). 
  <suite name="Suite" thread-count="2">
  <test name="Test">
    <classes>
      <class name="TestClassSample"/>
    </classes>
  </test>
</suite>

Compile the test case using javac.
/work/testng/src$ javac TestRetry.java TestClassSample.java

Now, run the testng.xml, which will run the test case defined in <test> tag. As you can see the tests grouped under name betaTest are skipped.
/work/testng/src$ java org.testng.TestNG testng.xml

Verify the output.

===============================================
Suite
Total tests run: 4, Passes: 0, Failures: 1, Skips: 0, Retries: 3
===============================================


You can see that test2 was run 3 times and its been marked failed only at the last run.