Mutant Score & Analysis Example – Mutation Testing in Software Testing

Software Testing Automation Testing Testing Tools

Mutation Testing, aka program mutation, mutation analysis, or fault-based testing, is a type of software testing where testers introduce some minor changes in a source code to check the error detection capability of the source code.

Test cases that fail to execute are considered ideal. If the test case passes, that means there is some form of error in the code. Entering minor changes in the code is known as modifying, and that modified version of the source code is known as a mutant. If the test cases passed, testers say that "the mutant lived or survived." The goal here is to kill all mutants.

Mutation testing also aids in improving the test quality of the defined test cases/suites. Killing more mutants will improve the quality of the tests further.

Mutation Testing: Important Concepts to Remember

Following are some of the concepts that you will often come across while performing mutation testing.

Mutant

Mutant aka mutant programs are a mutated or modified version of a program's source code. If you execute test data through a mutant, you will get different results from the real source code.

Three types of mutants:

Killed mutants − A mutation test will kill some mutants during its execution. You will come across killed mutants when you run the mutation testing on the original or mutated version of a source code.
Survived mutants − As mentioned above, some mutated code remains alive even after running test data. The goal is to rewrite the code to kill these survived mutants, aka live mutants.
Equivalent mutants − These types of mutants are closely related to live mutants as they remain alive after executing test data. However, unlike live mutants, equivalent mutants share the same similarities as the source code while having different syntax in some cases.

Mutation Operators or Mutators

Mutators are the key players in the mutation testing process. They decide the type of change to make in the source code. These changes in source code are also known as mutation rules or faults.

Mutation Score

Mutation score or mutation adequacy is a score assigned to several mutants present in a source code. Since equivalent mutants share similarities with the source code, you must skip them while calculating the mutation score. The aim here is to achieve a high mutation score.

Step by Step Process for Mutation Testing

Step 1 − This is where mutators have to develop several versions of mutants and introduce them to the source code. Each mutant created must contain a single fault. The goal of the testers here is to make the mutant fails, which ensures the efficiency of the test cases.

Step 2 − This is where the tests cases are tweaked to detect faults in a program. Then those cases are applied to the original and mutant programs.

Step 3 − In this step, mutators compare the original program with the mutant one to check for faults or errors.

Step 4 − If mutant programs and original programs generate different outputs, the test cases have killed the mutant. As a result, the test case passed for detecting differences with original and mutated programs.

Step 5 − If the original and mutant programs offer the same results, that means the mutant is still alive. When that occurs, mutators are directed to prepare more robust test cases until they kill all mutants.

How to Create Mutant Programs?

A mutation is a single syntax change in a program statement in which each mutant program must differ from the original program.

For example, If −

The original program is written as −

if (x<y)
   Print "Hello"
else
   Print "Hi"

Then for making it a mutant, you can do a minor mutation in the syntax, such as altering the syntax such as −

if(x>y)
   Print "Hello"
else
   Print "Hi"

There are several ways to generate a mutant program.

Firstly, you can replace an operand with a different operand or a constant value (replacing x with y or constant value= 5).
You can insert a new operator/s. if x==y, then replace == with >= or if x>=y, then insert ++ in the statement, making it x>=++y
You can modify the programmatic statements. Delete the 'else' in 'if else'. Delete the entire 'if else' and find out the program's behavior.

Some other examples of mutation

Replace GOTO statement
Replace return statement
Delete a statement
Replace logical connectors

Types of mutation testing

Below are the types of mutation testing mostly used by mutation operators or mutators −

Value Mutation − In this test, you can change a parameter or constant value by +/-1.
Statement Mutation − In statement mutation, you have to can delete or duplicate or rearrange statements in a code block.
Decision Mutation − In this mutation, you have to mutate a code that makes decisions. For instance, you can change > to <.

Advantages of Mutation Testing

Mutation Testing (MT) helps to cover a significant section of code's logic
Easily catch errors that often bypass
Evaluate and modify the test quality of a test suite
Helps in testing a particular part of a code, not limited to branches, statements, or paths

Disadvantages of Mutation Testing (MT)

It is a time and resource-consuming task as you have to deal with a large number of mutants generated for every code block
You have to counter check every live mutant

Tools for Mutation Testing (MT)

Some tools used for mutation testing include Stryker, PIT, Insure ++, and Jumble.

Some Tips for Mutation Testing

Regardless of how small a program is, the number of mutants is always going to be large. Therefore, you may need to use a subset of mutants.

You may have to appoint different mutation operators as per the application's programming languages, design, and specifications.

If you discover live mutants, it could mean two things- the mutant may be invalid, or the data set is inadequate.

Some automatic mutation testing's unit testing frameworks and tools include jester, pester and MuClipse, etc.

Conclusion

The role of mutation testing is to evaluate and avoid uncertainty in a program. Test-Driven Development (TDD) often helps in coding, but they can't always promise a lean code. Regardless of how well you engineer a code, they do not always work as per your expectations. Sometimes developers add codes without describing the expectations, which often puts the entire codebase at risk. Mutation Testing is used to catch those breaches in a TDD cadence.

Vineet Nanda

Updated on 30-Oct-2021 11:46:11

Previous Page Print Page Next Page