In this article, we will learn soak testing, its objectives, characteristics, what strategy to follow, what is the right time to begin soak testing, the test scenarios and some of its best practices, its advantages and disadvantages, challenges faced during soak testing, some tools and examples of soak testing.
Soak testing is a category of performance testing, in which the software product is tested under massive volume of load for extended period of time. Soak testing checks the behavior or performance of the product under production use. It tests the capability of the product to withstand massive load volumes for an extended period of time.
Soak testing is conducted at the system level to determine what would happen beyond the design expectations of the product. This testing is non-functional and is also referred to as endurance testing or longevity testing. When a software product is subjected to a high load for an hour, then it may wok fine. But when the same product is subjected to the same load for the whole day (24 hours), it may crash. Here, soak testing comes into use. Soak testing helps discover various issues in the product when it is subjected to continuous heavy traffic.
To determine the product’s behavior when subjected to heavy load for a long time.
To determine the product’s behavior when subjected to heavy load for a long time.
To determine the performance and capacity of the product.
To ensure the reliability and stability of the product.
To identify issues such as incorrect memory management, database connection issues, poor response time, etc.
In general, the duration of soak test depends on the time available for the project.
To need an extended period of time, the product must perform without any interruption.
It must cover all the conditions, agreed upon by the stakeholders.
In general, every product has a regular maintenance window time period. The gap between such periods is key to determine the scope of soak testing.
Like other performance testing, soak testing must be conducted while product is in the development stage. This testing is performed alongside functional testing. But this is never followed. Why? To manage project cost.
Functional testing gets the primary focus, and all other performance testing is performed close to the release of product. Generally, soak testing is performed just after load testing and before the product is released to the client. However, this includes a big disadvantage in terms of resolving issues in the product.
Any issue related to the performance of the product, found at a later stage of the project, is always difficult to resolve, and also involves major changes in the code. Sometimes, this might not be possible considering that release date of the product is close.
The best time to perform soak testing is on weekends so that the product can remain in running state for as long as a day or night. Moreover, it is advised to perform soak testing on time to discover and fix issues.
In long session soak testing, multiple day’s tasks are executed in a limited time frame. The number of business transactions in this limited time period must match or exceed the total number of transactions of the multiple days. The primary focus is the number of processed transactions. The most important activity involved in soak testing is to determine the available memory in the CPU and the required amount of memory for testing. The memory used at the start of the test and at the end of the test must be recorded. If required, memory usage of facilities such as JVM must also be recorded.
Some of the important guidelines to be followed before beginning soak testing are as follows −
Determine the load for which the product has to be soak tested.
Determine the time period for which the product is to be tested.
Determine the scenarios and testing environment details of the soak testing.
Record the resource consumption by database.
Record the server resource usage, such as CPU usage.
The test must run with realistic user concurrency.
Determine the risks involved in the testing. Prepare a backup plan for such emergency situations.
Consider an e-commerce website announces a festive sale, then the website might get loaded during the sale. In such situations, the website must be soak tested to avoid and deal with any unexpected failure.
At the end of the financial period, a bank’s website may experience heavy load continuously. Thus, the website must be soak tested to avoid and manage any unexpected failure.
It is always a good practice to test the software product with a load double its load handling capacity. For example, a website can handle 1000 users for 10 hours. Then, the website must be soak tested for 2000 users for 10 hours. This would help determine the behavior or response of the website under such abnormal conditions.
It is better to perform soak testing at during night, if long session soak testing is to be performed, then it is advised to do it on weekends. Non-working hours are most ideal for soak testing.
It is a good practice to analyze the risks and challenges associated with the soak testing, and also to create a recovery plan for the same.
It helps improve performance of the product.
It increases the product’s resistance.
It prepares the product to work under heavy load.
It enhances the product’s capability to withstand heavy continuous load.
The timeline of the project may get influenced by soak testing, as it is timeconsuming generally.
Resources may get swamped during testing due to high memory usage, because of a large number of users using the product.
Memory allocation − Memory leakages that often lead to memory crisis or manifest errors.
Database resource usage − Under some conditions, database cursors are unable to be closed, causing the entire system to stall.
Performance degradation − While ensuring that the response time after a long period of activities is as good as in the starting of the test, soak testing can also degrade performance.
Under some conditions, it may not be able to close connections between tiers of a multitiered system, which can stall some or even all the modules of the system.
Response time of some functions may degrade due to the inefficiency of the internal data structures during a long session soak test.
Soak testing is very time consuming, thus at times, it may get avoided due to shortage or unavailability of time.
The test environment for soak testing must be carefully chosen so that other tests performed on the product do not get effected. This may take place, because testing of the product under heavy load for a longer period of time could result in issues.
Soak testing demands automation tools, as the tests run for longer periods of time with massive number of users. Moreover, it also needs technical expertise.
In soak testing, it is not so easy to determine the required test volume.
testers must separate the test environment from the production environment, otherwise any error identified during testing can cause permanent data loss or data corruption.
At the end of the financial year, a bank’s system may witness a large number of transactions. This is occasional and unexpected, but the system must be able to handle this situation efficiently.
Staffs in any organization are off-duty on weekends, thus the system is expected to withstand this period on its own. So, soak testing would determine the capability of the system to function for a period longer than weekends-worth-of-continuous action.
A system is expected to witness 500 transactions per day, in general. Thus, it is necessary to test the system’s ability to perform beyond this general daily limit.
Suppose a system needs to process 50,000 logins, which represents seven days of activity. A 50–60-hour soak test can begin by Friday evening, and can be completed by Sunday night or Monday morning. Only such a test can help observe the degradation in the performance of the system.
Mobile apps, video games, etc. may be left in running state for a prolonged period of time, in different modes, for instance idling, paused at title screen, etc. to determine whether that app or game can handle continuous expected load.
LoadStorm − This tool from Customer Centrix enables managing the performance of the entire system and can present results in form of graphs. It is basically an open source tool; however, it can be upgraded at anytime to avail months of contracts. It generates HTTP load from the cloud based on the test plan and test environment. It can also be used in load testing and stress testing. Some of its benefits include cloud-based platform, recording of scripts, sophisticated control of scripts, in-depth analysis, and in-built consulting.
LoadRunner − This HP tool is widely used in soak testing as well as load testing. The results obtained from this tool are considered a benchmark in the global industry. Its basic idea is to record and replay users’ activities, and then generate the desired load on the system. It can simulate real-world activities and can determine the performance of the system by generating virtual load.
LoadUI − This is an open-source testing tool. This tool allows complex load testing and performance testing by simply dragging around the various components. This tool allows to create and update test cases while they are in running stage. Some of its powerful features include usability through its visual interface, intuitive design, flexibility, etc. It also enables making changes during the test.
OpenSTA − This is a distributed testing tool designed around COBRA. Its current toolset performs HTTP and HTTPS heavy load tests with performance measurements from Windows-32-bit platforms and all other versions.
In software engineering, soak testing is performed to determine whether the product under testing can withstand continuous heavy load. It is type of non-functional testing, and falls under the sub-category of performance testing. This testing helps determine the memory usage by the product under testing. Developers often perform soak testing right after load testing is completed. Soak testing is very time consuming and can continue for several days. Thus, it is often performed on weekends when no other tests are being performed.