STLC stands for Software Testing Life Cycle. STLC is a sequence of different activities performed by the testing team to ensure the quality of the software or the product.
STLC is an integral part of Software Development Life Cycle (SDLC). But, STLC deals only with the testing phases.
STLC starts as soon as requirements are defined or SRD (Software Requirement Document) is shared by stakeholders.
STLC provides a step-by-step process to ensure quality software.
In the early stage of STLC, while the software or the product is developing, the tester can analyze and define the scope of testing, entry and exit criteria and also the Test Cases. It helps to reduce the test cycle time along with better quality.
As soon as the development phase is over, the testers are ready with test cases and start with execution. This helps to find bugs in the initial phase.
STLC has the following different phases but it is not mandatory to follow all phases. Phases are dependent on the nature of the software or the product, time and resources allocated for the testing and the model of SDLC that is to be followed.
There are 6 major phases of STLC −
Requirement Analysis − When the SRD is ready and shared with the stakeholders, the testing team starts high level analysis concerning the AUT (Application under Test).
Test Planning − Test Team plans the strategy and approach.
Test Case Designing − Develop the test cases based on scope and criteria’s.
Test Environment Setup − When integrated environment is ready to validate the product.
Test Execution − Real-time validation of product and finding bugs.
Test Closure − Once testing is completed, matrix, reports, results are documented.
In this chapter, we will understand the factors of comparison between STLC and SDLC. Let us consider the following points and thereby, compare STLC and SDLC.
STLC is part of SDLC. It can be said that STLC is a subset of the SDLC set.
STLC is limited to the testing phase where quality of software or product ensures. SDLC has vast and vital role in complete development of a software or product.
However, STLC is a very important phase of SDLC and the final product or the software cannot be released without passing through the STLC process.
STLC is also a part of the post-release/ update cycle, the maintenance phase of SDLC where known defects get fixed or a new functionality is added to the software.
The following table lists down the factors of comparison between SDLC and STLC based on their phases −
|Environment Set up||
|Deployment/ Product Release||
The common objective of testing is finding bugs as early as possible. And, once the bugs are fixed, make sure it is working as expected and not breaking any other functionality.
To achieve these goals, seven basic principles are given for software testing −
Testing can show that defects are present but there is no way to prove that there is no defect in the product. Testing phases make sure that the application under test is working based on the given requirement and it helps to reduce the probability of undiscovered defects in the application. But, even if no defects are found, it does not mean that it is absolutely correct. We can assume that AUT is matching with our exit criteria and maintaining the requirements according to SRD.
100% coverage or testing of all combinations of inputs and possible combinations are not possible except of trivial cases. Instead of exhaustive testing, risk analysis and priorities are used to define the scope of testing. Here, most of the real time scenarios can consider including most probable negative scenario as well. This will help us track the failure.
Testing activities should start as soon as possible and be focused on defined objectives in Test Strategy and expected results. Early stage of testing helps to identify Requirement Defect or design level discrepancies. If these types of bugs are captured in initial stage, it helps us save time and is cost-effective too. The answer to why testing should start at an early stage is very simple – as soon as the SRD is received, the testers can analyze the requirement from the testing perspective and can notice a requirement discrepancy.
Based on previous product defect analysis, it can be said that most of the defects are identified from small set of modules which are critical for application. These modules can be identified based on complexity, different system interaction or dependency on different other modules. If testers can identify these crucial modules, they can focus more on these modules to identify all possible bugs. In a study, it is found that 8 out of 10 defects are identified from 20% functionality of AUT.
What is pesticide paradox – if pesticides are frequently used on crops, there comes when the insects develop a certain kind of resistance and gradually the pesticides thus sprayed seem to be ineffective on the insects.
The same concept is applicable on testing also. Here, insects are bugs while pesticides are test cases that are used to run again and again. If the same sets of test cases are executed again and again, these test cases become ineffective after certain timeframe and the testers will not be able to identify any new defect.
To overcome these conditions, test cases should be revised and reviewed time to time and new and different test cases can be added. This will help in identifying new defects.
This principle states that two different type of application can’t be tested using same approach until both applications are of same nature. For example, if a tester uses the same approach for Web Based Application and Mobile Application, that is completely wrong and there is high risk of poor quality of product release. Testers should use different approaches, methodologies, techniques and coverage for different types and nature of applications.
This principle states finding defects and fixing them until the application or system is stable, is time consuming and also eats up on the resources. Even after fixing 99% of the defects, there is a high risk of unstable application. The first essential thing is to verify the stability of the application and the prerequisites of the environment. If these two conditions fulfill, only then we can start with the detailed testing.
Requirement Analysis is the first phase of STLC and it starts as soon as the SRD/SRS is shared with the testing team. Let us consider the following points to understand the Requirement Analysis in STLC.
The entry criteria of this phase is the provision of SRS (Software Requirement Specification); it is also recommended that the application architecture is handy.
In this phase, the QA team analyzes at a higher level what to test and how to test.
The QA team follows up with various stakeholders like Business Analyst, System Architecture, Client, Test Manager/Lead in case any query or clarification is required to understand the requirement.
Requirements may be functional or non-functional like performance, security, usability, etc. or both functional and non-functional.
The exit criteria of this phase is to complete the RTM document, automation feasibility report and a list of questions if applicable to be more specific on the requirements.
There are three major activities that are performed by the QA team in this phase. The activities have been described below.
The QA team identifies the scope of testing at high levels and divides into various functional modules. The team also identifies the types of testing required to perform – smoke testing, sanity testing, functional testing, regression testing, etc. The QA team analyzes the prerequisites and the environment details where testing is supposed to be performed. The team gathers details about the testing priorities and lays focus on the sequence of modules to be validated. It also identifies the requirement defects if modules are contradicted and functionality is not getting carried over along with other modules.
Requirements tracing is a process of documenting the links between the requirements and the work products developed to implement and verify those requirements. The RTM captures all the requirements at the Requirement Analysis along with their traceability in a single document. All of this is delivered at the conclusion of the life cycle.
The Matrix is created at the very beginning of a project as it forms the basis of the project's scope and deliverables that will be produced.
The Matrix is bidirectional, as it tracks the requirement forward by examining the output of the deliverables and backward by looking at the business requirement that was specified for a particular feature of the product.
In the requirement phase, the QA team analyzes the scope of automation for regression testing. If automation is added in scope, the team decides which tool can be used, what functionalities will be covered as automation, the time-frame and the resource allocation involved for automation development. Once this analysis is completed, the QA team provides the Automation Feasibility Report to different stakeholders to provide signoff.
In this chapter, we will see the Entry and Exit Criteria at different levels in STLC. The following points need to be considered to understand the criteria.
Ideally, the QA team does not proceed with the next phase until the exit criteria of the current phase meets. The entry criteria should include the completion of exit criteria of the previous phase.
In real time, it is not possible to wait for the next phase until the exit criteria is met. Now, the next phase can be initiated if the critical deliverables of the previous phase have been completed.
In each phase of STLC, the entry and exit criteria should be defined.
Entry Criteria for STLC phases can be defined as specific conditions; or, all those documents which are required to start a particular phase of STLC should be present before entering any of the STLC phase.
Entry criteria is a set of conditions that permits a task to perform, or in absence of any of these conditions, the task cannot be performed.
While setting the entry criteria, it is also important to define the time-frame when the entry criteria item is available to start the process.
For Instance, to start the Test Cases development phase, the following conditions should be met −
Exit Criteria for STLC phases can be defined as items/documents/actions/tasks that must be completed before concluding the current phase and moving on to the next phase.
Exit criteria is a set of expectations; this should be met before concluding the STLC phase.
For Instance, to conclude the Test Cases development phase, following expectations should be met −
Acceptance criteria means the expected behavior of a functionality, module and application as listed in the requirement documents. It is verification stages/checkpoints to determine whether or not the software system has met the requirement specifications. The main purpose of this test is to evaluate the system's compliance with the business requirements and verify if it has met the required criteria.
Acceptance Criteria is a set of statements, that mentions clearly about the expected pass/fail result. Acceptance criteria specifies both functional and non-functional requirements. These requirements represent “conditions of satisfaction or expected behaviour.” There is no partial acceptance; either a criterion is met or it is not met.
These criteria defines the boundaries and parameters of a functionality/module and determines whether functionality/module is completed and is working as expected.
High Level Acceptance criteria is mentioned at the Test Plan Level. The acceptance criteria is converted to a list of points to be verified or expected results at the test cases level.
Acceptance criteria is defined on the basis of the following attributes −
A test plan outlines the strategy that will be used to test an application, the resources that will be used, the test environment in which testing will be performed, and the limitations of the testing and the schedule of the testing activities. Typically, the Quality Assurance Team Lead will be responsible for writing a Test Plan.
A Test Plan includes the following.
The following points need to be considered for Test Planning in STLC.
Ideally, the Test Analyst (Lead)/the Manager prepares the Test Strategy/Test Plan Document.
Analysis is more focused on application related data/information.
It is the first phase of actual testing tasks.
This phase answers “WHAT is to be tested” and “WHAT RESOURCES are required to test”.
The basic entry criteria of this phase is provision of Requirement Documents (updated version of unclear/missing/clarified requirements) along with Requirement Traceability Matrix.
If automation is in scope, Automation Feasibility Report should be prepared before entering in this phase.
The exit criteria of this phase is completion of Test Strategy/Test Plan Document and Test effort Estimation document.
The main objective of this phase is to prepare a Test Plan/Test Strategy document. It includes three major aspects – Scope of Deliverables, Effort estimation and Resource Plan.
Following activities need to be performed to conclude over the scope of deliverables −
Estimation is the process of finding an estimate, or approximation, which is a value that can be used for some purpose even if input data may be incomplete, uncertain, or unstable.
Estimation determines how much money, effort, resources, and time it will take to build a specific system or product. Estimation is based on −
The four basic steps in Testing Estimation are −
Resource plans are the key element in testing phases. These plans are inversely proportional to the time taken by the testing team to complete a particular task. Increasing the number of resources will decrease the number of days of completion for a certain limit after that it will be saturated and increasing the resource will not have much impact and might not lead to a decrease in the completion days.
A Resource Requester, usually a project manager, creates resource plans to ask for resources, track efforts and costs. A Resource manager can modify and approve resource plans before the plans are used.
The normal workflow for a resource plan is −
Once the Test Plan is ready, the QA Team initiates the development of test cases. The main objective of this phase is to prepare test cases for an individual unit. These functional and structural test cases cover the functionality, points of verification and validation mentioned in the Test Plan.
The following points need to be considered for Test Case Development in STLC.
In this phase, the QA team writes the test case with a stepwise approach. The Test Case is then signed off by a Business Analyst after reviewing or rework on test cases in case modifications are required.
Once test cases are ready, the QA team prepares the Test Data based on preconditions.
The entry criteria of this phase is that the activities in test planning should be finished and the test plan should be ready.
The exit criteria of this phase is that the test cases should be signed off, the test data should be ready and the test scripts prepared if automation is in scope.
Test cases should be mapped with the Requirement Traceability Matrix to follow up on coverage of requirements if anything is missed.
Following are the three activities that are carried out in the Test Case Development phase −
Scenarios ease the testing and evaluation of a complex system. The following strategies help in creating good scenarios −
Enumerate possible users, their actions and objectives.
Evaluate users with hacker's mindset and list possible scenarios of system abuse.
List the system events and how does the system handle such requests.
List benefits and create end-to-end tasks to check them.
Read about similar systems and their behavior.
Studying complaints about competitor's products and their predecessor.
A test case is a document, which includes test data, preconditions, expected results and post conditions, developed for a particular test scenario in order to verify compliance against a specific requirement.
Test Case acts as the starting point for test execution. After the a set of input values is applied; the application has a definitive outcome and leaves the system at some end point which is also known as execution post condition.
Test Data is used to execute the tests on test ware. Test data needs to be precise and exhaustive to uncover the defects. To accomplish these three objectives, it is followed by a stepwise approach as given below −
The following diagram shows the different activities that form part of Test Case Development.
Test Environment consists of elements that support test execution with software, hardware and network configured. Test environment configuration must mimic the production environment in order to uncover any environment/configuration related issues.
The following points need to be considered in a Test Environment Setup.
It is a combination of hardware and software environment on which the tests will be executed.
It includes hardware configuration, operating system settings, software configuration, test terminals and other support to perform the test.
It is the most crucial aspect of the testing process. Unavailability or faulty environment setup can ruin all the testing efforts.
Practically, the QA team cannot start actual work without having the right environment to test.
It is and independent activity and can be started along with test case development.
Most generically, this activity is performed by developers in technical aspects while requirement conditions can be done by Customers/Business Analysts.
Readiness of the test environment can be validated by smoke testing, and performed by the QA team.
Ideally, we can say that the Entry Criteria of this phase is the provision of Test Plan, readiness of Smoke Test cases and preparation of test data.
The exit criteria of this phase is that the test environment should be ready and smoke testing should be performed successfully with expected results.
The following activities are performed in this phase.
Following factors play an important role for the designing of a test environment.
Determine if test environment needs archiving in order to take back-ups.
Verify the network configuration.
Identify the required server operating system, databases and other components.
Identify the number of license required by the test team.
Analyze the environment setup requirements and prepare a list of software and hardware requirements for the setup. Get the official confirmation for setup of the test environment and configure to access the test environment.
Once the environment is set up and the QA team has the access to it, a quick round of smoke testing should be performed for validation of test environment build stability. If the results are as expected, the QA team can move to the next phase else point out the discrepancies and wait for deployment after fixes.
Test execution is the process of executing the code and comparing the expected and actual results. Following factors need to be considered for a test execution process −
The following points need to be considered for Test Execution.
In this phase, the QA team performs actual validation of AUT based on prepared test cases and compares the stepwise result with the expected result.
The entry criteria of this phase is completion of the Test Plan and the Test Cases Development phase, the test data should also be ready.
The validation of Test Environment setup is always recommended through smoke testing before officially entering the test execution.
The exit criteria requires the successful validation of all Test Cases; Defects should be closed or deferred; test case execution and defect summary report should be ready.
The objective of this phase is real time validation of AUT before moving on to production/release. To sign off from this phase, the QA team performs different types of testing to ensure the quality of product. Along with this defect reporting and retesting is also crucial activity in this phase. Following are the important activities of this phase −
The real validation of product / AUT starts here. System Integration Testing (SIT) is a black box testing technique that evaluates the system's compliance against specified requirements/ test cases prepared.
System Integration Testing is usually performed on subset of system. The SIT can be performed with minimum usage of testing tools, verified for the interactions exchanged and the behavior of each data field within individual layer is also investigated. After the integration, there are three main states of data flow −
Note − In SIT testing, the QA team tries to find as many defects as possible to ensure the quality. The main objective here is finding bugs as many as possible.
A software bug arises when the expected result doesn't match with the actual result. It can be an error, flaw, failure, or fault in a computer program. Most bugs arise from mistakes and errors made by developers or architects.
While performing SIT testing, the QA team finds these types of defects and these need to be reported to the concerned team members. The members take further action and fix the defects. Another advantage of reporting is it eases the tracking of the status of defects. There are many popular tools like ALM, QC, JIRA, Version One, Bugzilla that support defect reporting and tracking.
Defect Reporting is a process of finding defects in the application under test or product by testing or recording feedback from customers and making new versions of the product that fix the defects based on the client’s feedback.
Defect tracking is also an important process in software engineering as complex and business critical systems have hundreds of defects. One of the most challenging factors is managing, evaluating and prioritizing these defects. The number of defects gets multiplied over a period of time and to effectively manage them, defect tracking system is used to make the job easier.
Once defect is reported and logged, it should be mapped with the concerned failed/blocked test cases and corresponding requirements in Requirement Traceability Matrix. This mapping is done by the Defect Reporter. It helps to make a proper defect report and analyze the impishness in product. Once the test cases and requirements are mapped with the defect, stakeholders can analyze and take a decision on whether to fix/defer the defect based on priority and severity.
Re-testing is executing a previously failed test against AUT to check whether the problem is resolved. After a defect has been fixed, re-testing is performed to check the scenario under the same environmental conditions.
During re-testing, testers look for granular details at the changed area of functionality, whereas regression testing covers all the main functions to ensure that no functionalities are broken due to this change.
Once all defects are in closed, deferred or rejected status and none of the test cases are in progress/failed/no run status, it can be said that system integration testing is completely based on test cases and requirement. However, one round of quick testing is required to ensure that none of the functionality is broken due to code changes/ defect fixes.
Regression testing is a black box testing technique that consists of re-executing those tests that have had an impact due to code changes. These tests should be executed as often as possible throughout the software development life cycle.
Final Regression Tests − A "final regression testing" is performed to validate the build that has not undergone a change for a period of time. This build is deployed or shipped to customers.
Regression Tests − A normal regression testing is performed to verify if the build has NOT broken any other parts of the application by the recent code changes for defect fixing or for enhancement.
Following block diagram shows the important activities performed in this phase; it also shows the dependency from the previous phases −
Defect Life Cycle, also known as Bug Life Cycle, is the journey of a defect, the cycle which a defect goes through during its lifetime. It varies from organization to organization and also from project to project, as it is governed by the software testing process and also depends upon the tools used.
The following diagram shows the workflow of a Defect Life Cycle.
Following are the different states of a Defect Life Cycle.
New − Potential defect that is raised and yet to be validated.
Assigned − Assigned against a development team to be addressed.
Active − The Defect is being addressed by the developer and investigation is under progress. At this stage, there are two possible outcomes – Deferred or Rejected.
Test / Fixed / Ready for Retest − The Defect is fixed and ready for testing.
Verified − The Defect that is retested and the test has been verified by QA.
Closed − The final state of the defect that can be closed after the QA retesting or can be closed if the defect is duplicate or considered as NOT a defect.
Reopened − When the defect is NOT fixed, QA reopens/reactivates the defect.
Deferred − When a defect cannot be addressed in that particular cycle it is deferred to future release.
Rejected − A defect can be rejected for any of the three reasons – duplicate defect, NOT a Defect, Non Reproducible.
Defects are classified from the QA team perspective as Priority and from the development perspective as Severity (complexity of code to fix it). These are two major classifications that play an important role in the timeframe and the amount of work that goes in to fix defects.
Priority is defined as the order in which the defects should be resolved. The priority status is usually set by the QA team while raising the defect against the dev team mentioning the timeframe to fix the defect. The Priority status is set based on the requirements of the end users.
For example, if the company logo is incorrectly placed in the company's web page then the priority is high but it is of low severity.
A Priority can be categorized in the following ways −
Low − This defect can be fixed after the critical ones are fixed.
Medium − The defect should be resolved in the subsequent builds.
High − The defect must be resolved immediately because the defect affects the application to a considerable extent and the relevant modules cannot be used until it is fixed.
Urgent − The defect must be resolved immediately because the defect affects the application or the product severely and the product cannot be used until it has been fixed.
Severity is defined as the impishness of defect on the application and complexity of code to fix it from development perspective. It is related to the development aspect of the product. Severity can be decided based on how bad/crucial is the defect for the system. Severity status can give an idea about the deviation in the functionality due to the defect.
Example − For flight operating website, defect in generating the ticket number against reservation is high severity and also high priority.
Severity can be categorized in the following ways −
Critical /Severity 1 − Defect impacts most crucial functionality of Application and the QA team cannot continue with the validation of application under test without fixing it. For example, App/Product crash frequently.
Major / Severity 2 − Defect impacts a functional module; the QA team cannot test that particular module but continue with the validation of other modules. For example, flight reservation is not working.
Medium / Severity 3 − Defect has issue with single screen or related to a single function, but the system is still functioning. The defect here does not block any functionality. For example, Ticket# is a representation which does not follow proper alpha numeric characters like the first five characters and the last five as numeric.
Low / Severity 4 − It does not impact the functionality. It may be a cosmetic defect, UI inconsistency for a field or a suggestion to improve the end user experience from the UI side. For example, the background colour of the Submit button does not match with that of the Save button.
A check against the test exit criteria is very essential to claim that the testing is now complete. Before putting an end to the test process, the product quality is measured against the test completion criteria.
The entry criteria of this phase is that the execution of the test case is complete, test results are available and the defects report is ready.
The criteria for test completion includes the following −
The exit criteria of this phase is the provision of test closure reports and preparation of matrices which are later signed off by the client.
Let us now discuss the activities involved in the closure of Test Cycle.
Test completion reporting is a process, whereby the test metrics are reported in summarized format to update the stakeholders. This enables them to take an informed decision.
Test Completion Report contains the following information.
A good Test Completion Report indicates quality, measures outstanding risks, and identifies the level of a tested software.
Upon completion of testing, various matrices are collected to prepare the test reports. The criteria for preparing the reports includes the following −
While executing a test case, re-testing defects and performing regression test case, Test results articulate should be saved and can be produced along with the test cycle closure documents to support the completion of test execution.
Articulates can be screenshots, database query results, recording, log files, etc.