Standard Deviation (SD)

Standard Deviation is a key metric in performance test result analysis which is related to the stability of the application. 

Definition:

The Standard Deviation is a measure of how response time is spread out around the Mean. Simply say, the smaller the Standard Deviation, the more consistent the response time.

Importance of Standard Deviation in Performance Testing

Standard Deviation in your test tells whether the response time of a particular transaction is consistent throughout the test or not? The smaller the Standard Deviation, the more consistent transaction response time and you will be more confident about particular page/request. Delivering a consistent experience to the end-user is just as important as delivering a fast and responsive experience. Let’s take an example:

Transaction Name:        RT          RT        RT        RT        RT        Avg    SD     90th %ile

                                      (I1)            (I2)        (I3)    (I4)        (I5)

Login                         4            6            3            4            8        5        2            6

Search                         3            2            15           1           4        5        5.7        4

Logout                         5            5            6            4            5        5        0.7        5

                    where:

RT         = Response Time

I            = Iteration

Avg       = Average Response Time

SD        = Standard Deviation

In the above example:

• Averages for all the transactions are the same. You cannot say the test results are good on the basis of average response time because averages are considered useless in Performance testing.
• The 90th percentile of “Search” transaction is better than the other two, but you can see I3 has 15 seconds response time. It is true we do consider percentile value as an important metric, but not alone. You also need to check how much response time is deviating.
• “Logout” transaction having lowest Standard Deviation (0.7) it shows response times are more consistent than other two and it is true we can see very less deviation in “Logout” response time 5, 5, 6, 4, 5. Also, its 90th percentile is 5.
• So, we got our best performer (Logout) and need to investigate the other two requests (Login and Search) for tuning purpose.

Calculation of Standard Deviation:

how Standard Deviation is calculated because as a performance tester you will be looking for a tool that calculates quick and correct Standard Deviation and save your time.     

Standard Deviation calculation steps:

1. Calculate the Mean (the simple average of the numbers)
2. Subtract the Mean from each number and square the result
3. Add up all the values then divide by N-1
4. Take the square root of that. It’s your Standard Deviation

Illustration of Search Transaction response time:

Step 1:     Mean = (3 + 2 + 15 + 1 + 4)/5 = 5

Step 2:    (3-5)2  = 22 = 4;

                (2-5)2  = 32 = 9;

                (15-5)2  = 102 =100;

                (1-5)2  = 42 =16;

                (4-5)2  = 12 =1;

Step 3:    (4 + 9 + 100 + 16 + 1)/4 = 32.5

Step 4:      √32.5 = 5.7

Qus :- SD : 30 sec and Mean Response Time is 40 Sec.

Ans :  Here A transaction may have calculated Mean Response Time of 40 Sec  but SD of 30 Sec. This means that end user has High Chances of Response Time as Low as 25 sec and High as 55 Sec for same activity.