Performance Testing Types

 What is a Baseline test?

Ans: We can run Baseline test of an application, collect and analyze the results, then modify the several indexes on a SQL server DB and run the same test again, using previous results to determine whether or not the new results were better, worse or the same.

Bench

How are benchmark and baseline tests different?

Ans: Benchmark Testing : It is a process of capturing your system performance against an industry standards that is given by some other organization.

This info can be used as a reference point when changes in the future are made to applications. On the other hand, benchmark tests are the process of comparing the performance of your system against industry standards giving by other organizations. For example, run baseline tests on applications, analyze the collected results and modify many indexes on the database of a SQL Server before running the identical test once more using the previous same results to find out whether or not new results are the same, worse or better.

What is Load Test?

Load Test is a type of non-functional test which verifies the performance of an application or system under a peak load condition. Load Test also validates the resource usage, stability and reliability of the software system under peak load.

What is Peak Load?

Peak Load is the highest load identified during a day, a month or a year depends upon the production data selection criteria. To understand the peak load refer to the below graph.

Figure 1: Production Data Graph to identify the Peak Load

This graph shows the number of active customers per day in the month of Jun. The highest number of active customers is 927 on 30th Jun. Hence 927 is the peak load for the load test.

For new application: Since a new application does not have any production data, so peak load needs to be predicted. The client or project business analyst (BA) confirms the expected peak load on the application. A performance tester can use the expected peak load to prepare the workload model for the load test.

Purpose of Load Test:

To Identify whether the application can handle the peak load

Observe the behaviour of the application in terms of response time

To check whether the resources (CPU, Memory and Disk) do not breach the defined performance limit

To identify if there is any bottleneck

Load Test is also conducted in a regression manner to identify the performance issue due to the weekly, fortnightly or monthly code releases.

Approach:

NFR document has some separate set of NFRs for the load test. These NFRs are related to the count of peak user load, response time, transactions per second etc. A performance tester designs the workload model using these NFRs and executes the test. Ideally, the duration of the load test is 1 hour (excluding ramp-up and ramp-down period). A typical load test user graph is:

Figure 2: A Typical Load Test Graph

This load test graph has a steady state of 1 hour along with 10 minutes ramp-up and 10 minutes ramp-down period. Therefore the test will run for 1 hour and 20 minutes. After completion of the test, a performance tester verifies the result against the defined load test NFRs.

What is a Soak Test or Endurance Test?

Soak Test is a type of non-functional test which helps to identify the Memory Leakage into the software system. Another name of the Soak Test is ‘Endurance Test’. In the soak test, a significant load is applied on the server for an extended period of time. Generally, the test duration of the soak test is in between 8 to 24 hours, but it may vary as per project requirement. A longer period performance test with an average load provides information about the behavior of Garbage collector and memory management.

Soak testing highlights issues such as:

• A constant degradation in response time when the system is run over time.
• Deteriorations in system resources that aren't evident during short runs, but will surface when a test is run for a longer time. For example, memory consumption and performance, free disk space, operating capacity, etc.
• Any periodical process that may affect the performance of the system, but which can only be detected during a prolonged system run. For example, a backup process that runs once a day, exporting data into a 3rd party system, etc.

How to calculate the Average Load?

The business analyst calculates the average load by referring to historical data. He may consider past days, weeks, months or years statistics to get the average number. He calculates the figure by using some mathematical formulae. To understand the average load calculation refers to the below graph. It is a typical graph; not a practical one:

Figure 1: A typical historical data for calculating the average load

Above graph shows the historical data of a month. On calculating the average of the given figures a number comes which is 100. Hence 100 is the average user load for the soak test.

In some cases, if data is not available for the calculation then 50%, 60% or 75% of peak load can be considered for the test. But before conducting the test get the confirmation on these volumes from the project team.

For a new application: Since a new application does not have any historical data, so the above-suggested % of peak load can be used for soak testing with proper approval from the project team. The purpose of the soak test does not change for a new or an existing application.

Purpose of Soak Test:

1. Verify the sustainability of the application
2. Check if there is any spike in the response time
3. Identify the Memory Leak
4. Check the behaviour of Garbage Collector
5. Identify if there is any bottleneck
6. Check the type of error due to the prolonged duration of the test and get the error percentage
7. Check whether the resources (CPU, Memory and Disk) do not breach the defined performance limit

Approach:

Many clients are not aware of the advantages of the Soak test. Hence they skip this test and do not provide any specific requirement for a soak test. Soak Test is one of the important performance tests and helps to detect memory leakage. NFR document must have a separate set of NFRs for the endurance test. These NFRs are related to the average user load count, response time, transactions per second etc. In the absence of an average load, conduct the test at 50%, 60% or 75% of peak load and analyse the behaviour of the application.

A performance tester designs the longer duration workload model using defined or calculated NFRs and executes the test. Ideally, the duration of the endurance test is 8 to 24 hours (excluding ramp-up and ramp-down period). A typical endurance test user graph is:

Figure 2: A Typical Soak (Endurance)Test Graph

This endurance test graph has a steady state of 23 hours along with 30 minutes ramp-up and 30 minutes ramp-down period. Therefore the test will run for 24 hours. After completion of the test, a performance tester verifies the result against the defined load test NFRs.

It is recommended to plan at least 2 soak tests in a single performance testing cycle and if both the tests have consistent results then only jump on to next type of performance test.

What is Spike Test?

Spike Test refers to a performance test which simulates a sudden high load on the server for a shorter period of time. This is a type of non-functional test which helps to identify the behaviour of an application or software system when an unexpected huge load arrives. The outcome of the spike test concludes whether the application can able to handle a sudden load or not. And, if it is then how much load?

The average load is considered as a base load. In some special cases, the peak load replaces the average load and becomes the base load.

Type of Spike Test:

Spike tests are of three types:

1. Constant Spike Test: A constant spike load applies to the server after a certain interval of time. In this type of spike test, all the spike have same height i.e. the same load.

Figure 1: Constant Spike Test Graph

2. Step-up Spike Test: A gradually increase spike load applies to the server after a certain interval of time. In this type of spike test, response time should measure at each spike and analyse how much it deviates from baseload response time?

Figure 2: Step-up Spike Test Graph

3. Random Spike Test: A random spike load applies to the server at random interval. Such a test is conducted for an application that frequently gets spikes in the production environment.

Figure 3: Random Spike Test Graph

How to calculate Spike Load?

The business analyst analyses the historical data and checks if any of the sudden spikes appeared in the past. As per his analysis, he suggests the number of users for spike load. He may also predict the numbers by analysing the company’s requirement. For example, if a company has a plan to conduct a flash sale or one-minute sale then he needs to calculate the spike load as per the registered and active user count. Although there are multiple methods to calculate the spike load which a business analyst must aware. This is not a task of performance tester. A performance tester has a responsibility to design the workload model as per spike test requirement.

For a new application: This is an optional test for a new application. Still, if the business wants to test the application with spike load then a performance tester can design the step-up spike test and note down the behaviour of the application.

Purpose:

1. Verify the sustainability of the application at the sudden huge load
2. Identify the deviation in the response time during spike load
3. Check the failure percentage of the transactions
4. Identify the type of error like 500, 504 etc.
5. Note the recovery time of the application in case application is down during spike
6. Identify if there is any bottleneck
7. Check whether the resources (CPU, Memory and Disk) do not breach the defined performance limit even during the spike period

Approach:

Spike test is an optional test in the performance testing world. Spike test scenario is rare in the production environment but unavoidable. If any huge spike identifies in the production then it must require an investigation. During the investigation, the same scenario simulates in the performance test environment and identifies the root cause.

To handle the same situation in the future, a spike test performs in the test environment and tune the application in case of any issue occurs.

A performance tester prepares the workload by referring to the spike test NFR. NFRs must have the defined Base Load and Spike Load so that an accurate workload can be created. Ideally, the duration of the spike test is 1 hour (excluding ramp-up and ramp-down period). The typical spike user graphs are shown above (Figure 1, Figure 2 and Figure 3).

In the result, application response time during the spike period is an important metric. Also, performance tester must give the attention to the break (failure) point of the application (if any) along with the type of errors and application recovery period. The type of errors provides an idea of whether the application is fully down or some of the weak functionalities are impacted due to sudden the spike and what is the reason for failure? The root cause analysis helps to identify the exact issue.

It is recommended to plan at least 2 identical spike tests in a single performance testing cycle and if both the tests have consistent results then only jump on to next type of performance test.