Data Structure
Networking
RDBMS
Operating System
Java
MS Excel
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
Physics
Chemistry
Biology
Mathematics
English
Economics
Psychology
Social Studies
Fashion Studies
Legal Studies
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
Mutation Testing in Software Testing – Mutant Score & Analysis Example
Mutation Testing
Mutation testing, another form of software testing, is a testing in which statement(s) of the source code is/are changed or mutated to determine whether the test cases can detect errors in the source code or not. It is conducted to ensure the quality of test cases in terms of its robustness that it must fail the mutated source code.
Mutation testing is not only done to determine the quality of existing software tests, but also to design new software tests. In mutation testing, a program is modified in small ways. It emphasizes helping testers develop effective tests and detect bottlenecks in the test data of the program.
In 1971, Richard Lipton first proposed mutation testing. Initially, due to its high cost, mutation testing was not much used. However, today it is widely used for several languages like Java and XML. It is a type of White box testing and can be applied to design models, specifications, databases, tests, and even XML. Mutation testing, being a typical structural testing technique. In case the real program and mutant program(s) yield uses the code’s structure to guide the test. It can be considered as a process of rewriting the source code in small ways to avoid redundancies in it.
One important thing the tester need to keep in mind is to keep the changes in the source code extremely small so that it does not affect the primary purpose of the program. It is also referred to as fault-based testing strategy because it involves a faulty program and is mainly used for unit testing.
Types of Mutation Testing
Value Mutations − In this testing, the values are modified to find errors in the program. A small value is modified to a larger value or vice-versa. Generally, constants are changed in value mutation testing.
Decision Mutations − In this testing, logical/arithmetic operators are modified to discover errors in the program.
Statement Mutations − In this testing, a statement is deleted or is replaced by another statement.
Procedure of Mutation Testing
Faults are created into the source code by introducing many versions called mutants. A mutant must have only one fault. The purpose is to cause such mutant versions to fail who show the efficacy of test cases.
Then, test cases are applied to the real program and to the mutant program. A test case must be appropriate and is adjusted to find faults in a program.
Compare the results obtained from the real program with that of the mutant program.
If the real program and mutant program(s) yield different output(s), the test case destroys the mutant. Therefore, the test case is suitable enough to find the change between the two programs- real and mutant.
Creating Mutant Programs
A mutation is a single syntactical modification made in the program statement. Each such mutant program must be different from the real program by exactly one mutation. For example,
| Real program | Mutant Program |
|---|---|
| If (x>y) | If (x<y) |
| print “Hello readers.” | print “Hello readers.” |
| else | else |
| print “How you doing?” | print “How you doing?” |
Changes to make in a mutant program
There are many techniques to generate mutant programs, such as −
| Operand replacement operators | Expression modification operators | Statement modification operators |
|---|---|---|
| The operand is replaced with another one or with the constant value. | An operator or an insertion of new operators is replaced in program statement. | Programmatic statements are changed so as to create mutant program. |
| For example, if (a>b) replace a and b values if (5>b) replace a with 5 | For example, if (a==b) replace == with >= and make the mutant program as if (a>=b) and adding ++ in the statement if (x==++y) | For example, Delete the else statement in an if-else statement. Delete the whole if-else statement to determine how a program act. Sample mutation operators:
|
Automation Mutant Testing
Mutation testing consumes a lot of time and is also much complicated to be executed manually. Thus, it is advisable to use automation tools and techniques to speed up testing process. These tools reduce expenditure of testing process as well. Some of the automation tools for mutation testing are −
Stryker − This open-source tool helps in mutation testing of.NET Core and .NET Framework projects. It enables testing the test cases by temporarily introducing bugs in the source code. It can control more than 30 supported mutations. Stryker speeds up testing by using code analysis and parallel test runner processes. This tool supports JavaScript, TypeScript, C# and Scala. Stryker uses smart and competent reports to discover surviving mutants and improve test effectiveness.
PIT Testing − PIT- process integration test- tool helps setup automated tests for process integration scenarios. The idea behind this tool is; processed messages are retrieved from the runtime of SAP Process Integration or SAP Process Orchestration System. The messages are stored in the database of the PIT system and can later be run on a different system, known as target system. These ran messages are collected from the target system after the run is completed. The, the results are compared against a reference message(s) from the source. This tool is a state of art in the field of mutation testing, offers standard test coverage for JAVA and JVM. This tool is fast, robust and easily integrates with modern test and build tooling. PIT testing is easy to use, actively developed and supported. It produces reports in an easy-to-read-and-understand format, integrating line coverage and mutation coverage information.
Concepts of Mutation testing
Mutants − It is the mutated of modified version of the source code, that contains small changes. When test data is executed through mutant, it yields different results as from the original source code. They are also referred to as mutant programs.
Survived Mutants − These are the mutants that are still alive after test data is executed through the original and mutated versions of the source code. Survived mutants must be destroyed, and are also referred to as live mutants.
Killed Mutants − These mutants are destroyed after mutation testing is completed. These are obtained from different results from the original and mutated source codes.
Equivalent Mutants − These mutants are closely related to live mutants as in, they are alive even after test data is ran through them. They are different from others as in, they have the same meaning as the original source code, irrespective of their possibly different context.
Mutation Score − Mutation score is the percentage of destroyed mutants with the total number of mutants −
Mutation Score = (Destroyed Mutants / Total number of Mutants) * 100
If mutation score is 100%, then test cases are considered to be mutation adequate. Studies and experiments have shown that mutation technique is an efficient way to measure the adequacy or appropriateness of test cases. However, a major drawback of mutation testing is high cost involved in creating mutants and executing test cases against that mutant program.
Advantages of Mutation Testing
A strong approach to gain high coverage of source program.
Capability to comprehensively test mutant program.
A good level of error detection.
Detects redundancies and ambiguities in source code, capability to detect all faults in the program.
Provides a most reliable and stable system.
Disadvantages of Mutation Testing
Extremely expensive as well as time-consuming due to large number of mutant programs that have to be generated.
An automation tool is a must for mutation testing due to its high complications.
Each mutation has the same number of test cases as of the original program; thus, a large number of mutant programs have to be tested with the actual test suite.
It involves source code changes, so cannot be applied to Black box testing.
An example of Mutation Testing
Consider a hospital website that allows users to register. It collects various information such as Date of Birth or age, etc. If a patient’s age is greater than 15, a physician is assigned to them as their main doctor. For this, it enables the ‘physician” function that finds available doctors. There might be some other functionality. Patients below 10 may get assigned a pediatrician, and so on. But we consider only the age-over-15 case. The code may look like −
Read age.
if (age>15)
Doctor = Physician()
End if.
Note that the code is specific to any language, thus would not run. It is just a pseudo code.
We aim to check if the dataset of the 4 values (14,15,0 and 5) is enough to discover all problems and redundancies in this code.
Now, how does mutation testing achieve this? First, create mutants- variations of the program. A mutant is simply a program written as a deviation. It comes with a self-seeded issue, e.g., arithmetic operator replacement, logical connector replacement, statement removal, etc. These replacements are referred to as mutation operators.
Conclusion
The best possible way to execute exhaustive testing is through mutation testing- the most comprehensive way to test any program. This technique checks the effectiveness and accuracy of the program to detect faults or errors in the system.
778 Views
