In this article, we will study scalability testing, different types of scalability testing, its importance, features and attributes, prerequisites, strategy, procedure, how is it different from stress testing, and its advantages and disadvantages.
Scalability testing is a type of non-functional testing. In this testing, the performance of the software product under testing is measured in terms of its capability to scale up or scale down the number of user request load and other performance attributes. The purpose of performing scalability testing is to ensure that the product has enough capabilities to handle an estimated increase in user traffic, data load, transaction counts frequency, etc. It basically tests the ability of the system to meet the growing demands. Scalability testing can be done at hardware, software and database level. Scalability testing tests the product’s ability to perform the function correctly with changes in size of the system.
Scalability testing determines the point at which the product stops scaling and identifies the reason behind it. The parameters used in scalability testing are different for each application. For example, scalability testing of a web page depends upon the number of users, CPU utilization, and network usage. While, scalability testing of a web server depends on the number of requests processed by the system per unit time.
The primary objective of scalability testing is to determine the product’s scaling with the growing workload.
Another objective is to determine the client-side condition and the end users’ experience of the product under various loads.
Another objective is to determine robustness on the server-side.
Response Time − It is the time taken by the software product to respond to a user’s request. Response time depends on the user load of the software product. In general, response time decreases with increasing user load. Software products with smaller response times are considered to be higher performance products. Response time determines the time taken by the server to respond or process a user’s request of the server under minimum load, threshold load and maximum load. It helps identify the point at which the software product fails. In clustered environments, where the product load is distributed among multiple servers, scalability testing needs to measure the limit to which the load balancer distributes the load among the servers. In such environments, the response time of each individual server must be carefully measured, and scalability testing needs to ensure that the response time of each server is same irrespective of the load on each server. For example, response time can be measured as the time taken by the user to enter the URL on a web browser to the time taken by the web page to load the content.
Performance − Performance is a measure of the number of users a software product can support without failing or busy standby condition.
Threshold Load − It is the number of requests processed by the software product withthe expected throughput.
CPU usage − It is the measure of CPU utilization while executing the instructions of the software product code. Its unit is Megahertz. In general, the more optimized is the product code, the lesser is the CPU usage observed. To minimize CPU usage, many enterprises use standard programming practices. For example, removing dead code in the product and reducing thread usage. One of the best practices to minimize CPU usage are sleep methods.
Memory usage − It is the measure of the memory consumed by the software product while performing a task. Its unit is Megabytes, Gigabytes, or Terabytes. A software product is considered scalable if it uses as much less memory as possible. Some of the best programming practices to reduce memory usage are avoiding the use of redundant loops, reducing the hits to the database, using cache, optimizing the usage of SQL queries, etc. For example, in case the storage space available for a specified number of users exceeds memory, developers will then be compelled to add additional database storage to compensate data losses.
Network usage − It is the measure of the bandwidth consumed by the software product. Its unit is bytes per second, frames received per second, segments received per second, segments sent per second, etc. Network usage aims at reducing network congestion. Usage of compression techniques are of the best programming practices that can help minimize network usage. For example, rather than following a queue mechanism to process users’ requests, developers may write code to process users’ requests whenever the database receives a request. Other attributes that are also tested during scalability testing are server requests, task execution time, loading time, time taken to fetch response from database, reboot time, printing time, session time, transactions per unit time, hits per unit time, requests per unit time, etc.
Define a repeatable process to conduct scalability test.
Select the criteria of the scalability test.
Decide the software tools required to perform the test.
Prepare the testing environment and configure the hardware needed to perform the test.
Prepare the test scenarios and scalability tests.
Prepare visual script, then verify it.
Prepare the scenarios of load test, then verify them.
Perform the test.
Evaluate the results obtained from the test.
Generate a report on the test.
Load distribution capacity − Analyze if the tool used for load testing can generate load from multiple devices, and whether it can be controlled and measured from a central point.
Operating System − Analyze the OS that the load generation agents and load test master are compatible with.
Memory − Analyze the memory requirements of the virtual user agent and the load test master.
Processor − Analyze the type of CPU needed by the virtual user agent and the load test master.
Before creating the tests, a test plan is developed. The attributes of a well-defined test plan are as follows −
Steps for scripts − The script of the test is prepared, and should have a detailed step to determine the actions performed by a user.
Run-time data − Test plan should determine the run-time data needed to interact with the software product.
Data-driven tests − For scripts with varying data at the run-time, one must have the proper knowledge of all the fields required by this data.
|Scalability Testing||Stress Testing|
|Tests the performance of the software product under user request load.||Test the behavior or performance of the product under unexpected heavy load.|
|Determines the response time of the product under heavy load.||Tests the performance of the product.|
|Load is gradually varied.||Load is abruptly varied.|
|Determines the threshold load of the product.||Determines whether the product can recover from a failure or not, and how.|
|Tests CPU usage as well as memory usage.||No such attributes are tested.|
|Tests the robustness on the server-side.||Tests robustness on the client-side.|
|Tests user experience under heavy data load.||Tests user experience under an unexpected variation in load.|
Upward scalability testing − This testing is performed to expand the number of users on a particular scale until the product crashes. Its primary objective is to identify the maximum capacity of a software product.
Downward scalability testing − This testing is performed when the load testing is not passed. In this testing, the number of users in a particular interval is decreased until the specified goal is achieved.
Scalability testing helps in tracking operative tool utilization.
The most important advantage is we can determine the web application limitations in terms of network usage, response time, network usage, CPU usage, etc.
Scalability testing helps control end-user experience under specific load.
It helps fix issues earlier which saves time as well as money.
Scalability testing makes the product more accessible.
Scalability testing includes a set of load tests with various hardware and software settings, and keeps the testing environment unchanged.
There is always a risk of missing functional faults.
The test environment cannot always be identical to the production environment.
Tests work well occasionally, but may fail in the testing stage due to wrong test scripts and test scenarios. This wastes a lot of time and money doing useless modifications.
In some cases, there is need of advanced tools and an exclusively experienced team for performance testing, which causes an over budget.
Scalability testing demands a high-level of expertise and testing skills and knowledge.
Scalability testing is a type of non-functional testing to determine whether a software product can scale up or down to the different attributes. Attributes used in scalability testing are different for each product.
The main purpose of scalability testing is to determine the point at which the product starts degrading under the maximum load, and helps take steps to ensure that the product is scalable enough so as to accommodate variations in internal applications, hardware, and database.
If done properly, scalability testing can help identify major bottlenecks in terms of performance of the software, hardware and database. Some of its major disadvantage includes its storage restriction, database size limits, network bandwidth limitations, and the buffer space.