How to make TestNG print a detailed message about a failure?

TestNGRest AssuredDynamic Programming

TestNG supports a lot of assertions. It has the org.testng.Assert class that extends the Java object class java.lang.object.

Whenever there is a failure, a user would want to get a detailed failure report in order to perform a root cause analysis. But, sometimes the displayed failure information is not sufficient and sometimes the user is supplied with the same type of failure information repeated every time. These kinds of issues can be handled in TestNG using the keyword verbose ="<int>".

TestNG supports verbose and allows the user to pass a value based on the extent of failure information they want to see. It supports the values from -1 to 10.

  • By default, it's value is set to "0" which would show the least possible information about a failure.

  • "-1" is the debug mode.

  • When verbose is set to "10", it would show all the relevant stats about the tests, methods, listeners etc.

In this article, we will compare two lists in TestNG when verbose is 0 and 10.

Scenario 1

In this scenario, we will keep verbose=0 and observe the output.

Approach/Algorithm to solve this problem −

  • Step 1 − Create a TestNG class called NewTestngClass.

  • Step 2 − Write a @Test method in the class as shown below.

  • Step 3 − Now create the testNG.xml for verbose=0 to run the TestNG classes.

  • Step 4 − Run the testNG.xml or run the testNG class directly in IDE or compile and run it using command line.

Example

Use the following code for the common TestNG class "NewTestngClass" −

src/ NewTestngClass.java

import org.testng.Assert;
import org.testng.annotations.*;
public class NewTestngClass {
   @Test
   public void testOne() {
      int Id = 1;
      int newId = 2 ;
      Assert.assertEquals(Id, newId);
   }
}

testNG.xml

This is a configuration file that is used to organize and run the TestNG test cases. It is very handy when limited tests are needed to execute rather than the full suite.

<?xml version = "1.0" encoding = "UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name = "Suite1" verbose="0">
   <test name = "test1">
      <classes>
         <class name = "NewTestngClass"/>
      </classes>
   </test>
</suite>

Output

java.lang.AssertionError: expected [2] but found [1]
Expected :2
Actual :1

at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:907)
at org.testng.Assert.assertEquals(Assert.java:917)
at NewTestngClass.testOne(NewTestngClass.java:12)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:133)
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:598)
at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:173)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:824)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:146)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.testng.TestRunner.privateRun(TestRunner.java:794)
at org.testng.TestRunner.run(TestRunner.java:596)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:377)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:371)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:332)
at org.testng.SuiteRunner.run(SuiteRunner.java:276)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1212)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1134)
at org.testng.TestNG.runSuites(TestNG.java:1063)
at org.testng.TestNG.run(TestNG.java:1031)
at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)

Scenario 2

In this scenario, we will keep verbose=10 and observe the output.

Approach/Algorithm to solve this problem −

  • Step 1 − Create a TestNG class called NewTestngClass.

  • Step 2 − Write a @Test method in the class.

  • Step 3 − Now create the testNG.xml for verbose=10 to run the TestNG classes.

  • Step 4 − Run the testNG.xml or run the testNG class directly in IDE or compile and run it using command line.

Example

Use the following code for the common TestNG class "NewTestngClass"

src/ NewTestngClass.java

import org.testng.Assert;
import org.testng.annotations.*;
public class NewTestngClass {
   @Test
   public void testOne() {
      int Id = 1;
      int newId = 2 ;
      Assert.assertEquals(Id, newId);
   }
}

testng.xml

This is a configuration file that is used to organize and run the TestNG test cases. It is very handy when limited tests are needed to execute rather than the full suite.

<?xml version = "1.0" encoding = "UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name = "Suite1" verbose="10">
   <test name = "test1">
      <classes>
         <class name = "NewTestngClass"/>
      </classes>
   </test>
</suite>

Output

...
... TestNG 7.4.0 by Cédric Beust (cedric@beust.com)
...

java.lang.AssertionError: expected [2] but found [1]
Expected :2
Actual :1
<Click to see difference>
at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:907)
at org.testng.Assert.assertEquals(Assert.java:917)
at NewTestngClass.testOne(NewTestngClass.java:12)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:133)
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:598)
at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:173)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:824)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:146)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.testng.TestRunner.privateRun(TestRunner.java:794)
at org.testng.TestRunner.run(TestRunner.java:596)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:377)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:371)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:332)
at org.testng.SuiteRunner.run(SuiteRunner.java:276)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1212)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1134)
at org.testng.TestNG.runSuites(TestNG.java:1063)
at org.testng.TestNG.run(TestNG.java:1031)
at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)
===== Invoked methods
NewTestngClass.testOne()[pri:0, instance:NewTestngClass@35fb3008]
=====
FAILED: testOne
java.lang.AssertionError: expected [2] but found [1]
at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:907)
at org.testng.Assert.assertEquals(Assert.java:917)
at NewTestngClass.testOne(NewTestngClass.java:12)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:133)
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:598)
at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:173)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:824)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:146)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.testng.TestRunner.privateRun(TestRunner.java:794)
at org.testng.TestRunner.run(TestRunner.java:596)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:377)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:371)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:332)
at org.testng.SuiteRunner.run(SuiteRunner.java:276)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1212)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1134)
at org.testng.TestNG.runSuites(TestNG.java:1063)
at org.testng.TestNG.run(TestNG.java:1031)
at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)
===============================================
test1
Tests run: 1, Failures: 1, Skips: 0
===============================================
===============================================
Suite1
Total tests run: 1, Passes: 0, Failures: 1, Skips: 0
===============================================

Analysis

When verbose is 10, the user can see in output, there are lots of details such as

  • total tests run, which test ran and status

  • total Suite run, suite name and test run details.

  • Apart from this, user can see TestNG version that ran.

  • Invoked method name and respective Test name along with error.

The above details are missing when verbose is 0.

raja
Updated on 09-Mar-2022 11:28:15

Advertisements