Requirement Gathering
Performance Test Requirements
According to Tom Gilb’s principle, “Projects without clear goals will not achieve their goals clearly.”
Test requirements define your task set and provide you the goals to measure the end result of your activity. In performance testing world, we often face situations where an application is given to test without test requirements. In such situation, Performance Engineer has to get the required information before starting the testing. Performance Engineer has to divide the Performance Testing Requirements Gathering activity in two phases, first what elementary information he/she needs to successfully complete the task and secondly, how to collect the desired information.
Per Req
Following are some of the question comes to a Performance Engineer’s mind before starting a Performance Testing project,
What is the type of application and its complete architecture description?
What are the known current as well as previous performance bottlenecks?
Which application scenarios need to be tested?
What will be the workload model?
What are the performance goals?
Answer of each of the above questions has significant impact on the results of a performance test. Performance test results are always depend on the test environment and replicating exact production environment is extremely important for successful performance testing. Moreover, you can never be fully confident about your testing strategy and performance optimization recommendations without prior complete knowledge of application architecture. Information of known performance bottlenecks will assists you to setup the right type of test (Load Test or Stress Test etc.) to reproduce the production bugs (you need to reproduce the production bugs to find out their root cause). You can never do 100% coverage of the application in performance testing and identification of the critical business scenarios will help you to find out the real issues in optimal time. Performance testing workload should be as much identical to production environment as possible. Assigning the right user mix to selected performance scenarios is absolutely important for accurate results. Last but not the least, Performance goals will define the success criteria of your activity and you can never produce quantitative results without clearly defined performance goals.
Once all the required performance test requirements are listed, next step is how to gather this information. You can contact to different stakeholders like Business Analysts, Marketing Team, Network Team, Development Team and Functional Testing Team to collect all the required information.
Best you can do by preparing a questionnaire and forward it to all stakeholders to get the maximum information from them. But what will you do if you don’t receive any useful information from all the contacted sources? Then you have no choice but to use your experience and expertise to extract maximum requirements. Of course one may not be able to list down all the required information by own but at least can get enough to successfully complete the activity. For Eg, if you are going to start testing an E-Commerce web application, an experienced performance engineer can guess what will be its potential scenarios (E.g. Browsing Catalogs, Searching for items and Adding Items to Cart etc.). One can also figure out Browsing catalog will be the mostly accessed and searching will be most performance intrusive scenario. A Performance Engineer can easily figure out an E-Commerce web application Response time should not be more than 3 seconds. So it’s not the end of world if you don’t get the required performance test requirements and you can still successfully complete the activity by using your experience.
What are the Non-Functional Requirements?
Non-functional requirements are the testing goals which are created especially for performance testing, security testing, usability testing, etc. It is a combined requirement for all types of non-functional test. Since PerfMatrix is a core performance testing site, so non-functional word will be specific to Performance Testing only. Some simple examples of Non-Functional Requirements are:
- The number of users handled by the application
- Page Response Time
- The number of requests processed by the application per unit time
- CPU Utilization
- Memory Utilization
- Error rate etc.
In non-functional requirement, some of the goals are time-bound like the response time of a page, request per second, resource utilization etc. whereas some of the goals are load bound like real-world user load, throughput etc.
Difference between functional and non-functional requirement:
The main difference in functional and non-functional requirement is that functional requirements are end-user result oriented and stressed over the correct output. For example: If a user uses a banking application and clicks ‘Account Balance’ link then the application must display the correct balance available in his account. On the other hand, the non-functional requirements are oriented to the performance of the system in terms of responsiveness and load-bearing capacity. For example: If 1000 users hit ‘Account Balance’ link at the same time then the application must respond to all the users within a defined time (say 3 seconds).
Purpose of NFR Gathering:
The project team or client sets the expectation for performance testing in the form of non-functional requirements. To collect the requirement, analyse them from performance testing perspective and finalise the quantitative NFRs; all these steps fall under the NFR gathering phase of PTLC (Performance Test Life Cycle). All the requirements are documented, categorized and concluded in the Non-Functional Requirement Document. The end result of this phase provides quantitative NFRs which helps to prepare a correct workload model during performance testing.
Accountability:
Performance Test Manager or Performance Test Lead has a responsibility to collect, discuss and finalise the Non-functional requirement.
Approach:
Once the performance testing scope is finalised in the risk assessment phase, then either of them (Performance Test Manager or Lead) has to schedule a meeting with the project team to understand the client’s expectation. He may get the requirement in layman term which may require thorough study and deep analyse to extract the testable NFRs. There is also a high probability not to get clear and reasonable requirements in one meeting so he needs to set-up multiple meetings with the project stakeholders. Once all the non-functional requirements are finalised and he gets the testable NFRs then these NFRs should be properly documented in the Non-Functional Requirement Document and get the approval from the project stakeholders.
How to proceed practically?
“How to get the performance requirements from a non-technical customer?” This is a valuable question that everyone has. In most of the cases, the non-functional requirements are either defined incompletely or they are more conceptually rather than quantitative. A Performance Test Manager/Lead needs to spend his time to understand the client’s expectation by asking the right set of questions and transforming the conceptual requirements into quantitative goals. He needs to play the role of a mediator who bridges the gap between the novice user language and performance testing terminologies.
It is the responsibility of Performance Test Manager/Lead to get the essence from the customer. One needs to quickly understand the customer and should start talking in their language. Don’t talk using performance testing jargons and make the customer feel that they are ignorant, which is not the case. Don’t expect that the customer will give the performance goals in a single shot. Rather, start the discussion with an overview and purpose of performance testing. The discussion can be started with questions like
- What is the expected user load they are looking for?
- How much the expected response time for a page?
- What are the types of performance test can be included? Explain each type of performance tests and their purpose.
These simple questions make the client comfortable to explain what he wants.
Some additional Tips:
Also, try to build a good rapport with the customer. During the meeting/call, educate the customer by explaining some core performance testing terms with some good examples, so that he can explain exactly what his expectations are. The most important thing which a performance manager/lead should always avoid, not to scare the client by asking lots of question in one shot or talking too much technical. The way of asking the requirement should be very polite to impresses the customer and get his confidence.
Once the NFRs are collected, draft them in Non-Functional Requirement Document along with the date and points discussed in the meetings. The final Non-Functional Requirement Document must be signed-off from all the project stakeholders.
Challenges:
In most of the cases, applications are new and hence a Performance Test Manager/Lead does not have any clue on the NFRs. Many times, the client pick some random numbers over the call and ask to perform NFT (non-functional testing) against those numbers and at the end of the test when such NFRs do not meet, client forced to modify the NFRs and pass the test at any cost.
Such performance tests are executed to showcase that performance testing had been done and the application performed very well in the test environment. But in reality, 82% of such applications are failed due to performance issue within 6 months or even in the less time period when user load increases in production. Hence key points are to analyse the NFR thoroughly, do not consider (agree) any random number as NFR and finalise the quantitative NFRs without any caveat.
Deliverable:
Non-functional requirement document has to draft once the scope is finalized. The outcome of each meeting may lead to multiple changes in the document. Non-functional requirement document should cover all the aspects along with major and minor changes in the requirement. Do not forget to mention the acceptable error percentage in the document because this is one of the criteria which will help to decide RAG status at the end of the testing.
Non-Functional Requirement Document Template:
Download the template of Non-Functional Requirement Document.
Example:
Let’s call PerfMate. In the previous phase (Risk Assessment) of the project, PerfMate has conducted the risk assessment and finalise the scope of performance testing. He got the approval from all the stakeholders on Risk Assessment document. Moving to the next phase, he starts to collect the non-functional requirement (specific to performance testing) and receives the following requirements:
- The application should be very fast.
- The response time of the application should be quick.
- The web server performance should be as high as possible.
- The application should support many users.
- The servers should not fail when a sudden load comes during sale and offer periods.
- The application should run without any failures for a long duration.
From a client point of view, he has given all the requirements and set his expectation, but from PerfMate perspective, he just got the conceptual requirements. Now, PerfMate explained to the client that the provided information is partial and cannot help him to define the performance testing goal. With a polite manner, he gained the customer’s confidence and convinced him to provide the quantitative requirements. He asked project stakeholders to answer some of his basic questions. At last, he managed to gather the following requirements in the next couple of days:
Question: How many types of users are using this application (GUI)?
Answer. 4 types of users
- Admin
- Seller
- End-user
- Call center employee
Question: What are the business scenarios of every user?
Answer: Business scenarios for each type of users are like:
Admin:
- To approve/reject a new seller
- To verify a newly added product and approve/reject it
Seller:
- To add a new product
- To delete the existing product
Buyer:
- To buy a product
- To cancel the order
Call Center Employee:
- To register the complain
Question: What is the AUT current and predicted peak user load for all its users’ actions over time?
Answer: Admin
- Current: 4
- Predicted: 10
Seller
- Current: 25
- Predicted: 100
Buyer
- Current: 438
- Predicted: 2500
Call Center
- Current: 8
- Predicted: 24
Question: What is the average of active user count (including all types of users) at a time?
Answer:
- Total: 304
- Admin: 3
- Seller: 15
- Buyer: 278
- Call Center: 8
Question: What could be the active user count during peak hour?
Answer:
- Total: 500
- Admin: 4
- Seller: 50
- Buyer: 438
- Call Center: 8
Question: Anytime during a day or month when average user count is suddenly increased?
Answer: 31st of every month there is a 1-minute sale from 09:00 to 09:01 and 10:00 to 10:01. During this period the active buyer count becomes three times i.e. 834 and active seller count becomes 23. None of the other (Admin/Customer Care) users count change.
- Total: 868
- Admin: 3
- Seller: 23
- Buyer: 834
- Call Center: 8
Question: What is the request count receive at Admin end?
Answer: 200 requests per hour
Question: What would be the response time NFRs for all the scenarios?
Answer: None of the pages should breach 3 seconds average response time NFR. This NFR is applicable for all type of users, scenarios, and pages.
<Rest of the figures are displayed in the below NFR table>
Question: What would be the server-side NFRs?
Answer:
- CPU utilization must not be more than 60%; except stress test.
- Pre, post and steady-state memory utilization difference must not be more than 15%; except stress test.
- Pre, post and steady-state disk utilization must not be more than 25%; except stress test.
Question: What is the size of the performance testing environment with respect to production?
Answer: 100% (Live like environment)
More or less PerfMate got the answers to all of his queries and noted down in the NFR document and share with the client for the approval. After getting approval on the NFR Document, he will start the preparation of the performance test strategy.
A typical NFR format prepared by PerfMate is:
Note: Do not consider the below table as a standard NFR format. It was made simple for understanding purpose; especially for beginners.
NFR ID Category Description Impact to
NFR01 Application The solution must be able to support 500 active users.
1. Admin: 4 1. Admin
(2 for seller and 2 for product approval)
2. Seller: 50 2. Seller
3. Buyer: 438 3. Buyer
4. Call Center: 8 4. Call Center
NFR02 Application The solution must be able to support the future volume of active users i.e. 2634
1. Admin: 10 1. Admin
2. Seller: 100 2. Seller
3. Buyer: 2500 3. Buyer
4. Call Center: 24 4. Call Center
NFR03 Application The solution must be performed well during the longer period of time with average volume. i.e. 304
1. Admin: 3 1. Admin
2. Seller: 15 2. Seller
3. Buyer: 278 3. Buyer
4. Call Center: 8 4. Call Center
NFR04 Application The solution must be able to support the spike load of the buyer and seller during the sale period.
1. Admin: 3 1. Admin
2. Seller: 23 2. Seller
3. Buyer: 834 3. Buyer
4. Call Center: 8 4. Call Center
NFR05 Application Admin gets an average of 200 requests per hour every time. 1. Admin
NFR06 Application The number of orders:
1. Peak Hour volume: 1340
2. Sale Hour Volume: 2830
3. Future Volume: 7500
4. Average Volume: 600
Note: 4% of the users cancel the order in every scenario. 1. Buyer
NFR07 Application Sellers add an average of 180 products per hour and delete 18 existing products every hour 1. Seller
NFR08 Application The call center employees get 40 complains per hour 1. Call Center
NFR09 Application The response time of any page must not exceed 3 seconds except stress test 1. Admin
2. Seller
3. Buyer
4. Call Center
NFR10 Application The error rate of transactions must not exceed 1% 1. Admin
2. Seller
3. Buyer
4. Call Center
NFR11 Server The CPU Utilization must not exceed 60% 1. Web
2. App
3. DB
NFR12 Server The Memory Utilization must not exceed 15% (Compare pre, post and steady state memory status) 1. Web
2. App
3. DB
NFR13 Server The disk Utilization must not exceed 15% (Compare pre, post and steady state memory status) 1. Web
2. App
3. DB
NFR14 Server There must not any memory leakage 1. Web
2. App
3. DB
NFR15 Application Buyers order at the average rate of 1. Buyer
1. Peak Hour Rate: 3.06 products per hour
2. Sale Hour Rate: 3.39 products per hour
3. Future Volume: 3 products per hour
4. Average Volume Rate: 2.15 products per hour
=======
General Information
1. What is the current project timeline to begin and close testing activities? ie. starting and completion date
2. Is the application functionality stable and its functional testing is completed?
3. We have observed that in some cases firewall affects the results in load testing, do we need to get firewall clearance before starting load testing?
4. Please provide access credentials/URL of the application?
5. Any preference on Performance Tools? E.g. LoadRunner, JMeter
6. What type of performance testing should be performed? E.g. Load Test, Stress Test, Soak Test, Spike Test
7. What are the goals of the performance testing activity?
E.g.
Evaluate System against performance criteria
Discover what parts of the system perform poorly and under what conditions
Compare two platforms with the same software to see which performs better
8. What will the acceptance criteria for each performance test?
E.g.
All user transaction should pass with response time not more than 5 seconds
At least 95% of the user transaction should be successfully completed
Architectural Information
9. What is the type of application? E.g. Client Server, Web Based, Mobile App
10. In which technology/Platform the system is developed?
E.g. J2EE, .Net, PHP, Silverlight, Ruby, SAP, Any other
11. Which data base is used for this system? E.g. Oracle, MySQL, SQL Server
12. Which Application server is running with the system? E.g. Tomcat, IIS, WebSphere
13. What does the target system (hardware) look like (Please specify all servers and network appliances configurations and their interaction mechanism)?
LAN/WAN details
Terminal servers
Bandwidth link
Load Balancing techniques
Batch Transactions
Disaster recovery
14. Do you have traffic monitoring tool deployed on web server? E.g. Google Analytics, New Relic
15. Is there any known issue(s) in this application?
E.g.
Memory lock
Higher CPU and Memory utilization
Unexpected growth in daily visitors
More response time which leads to time out error
16. What is the protocol between client and server? E.g. http, https, TCP, FTP
17. For a web application, is the client browser version dependent? E.g. Application runs only on IE-8
18. Will you provide us separate test environment to do a performance test run?
Note: Its strongly recommended test environment should be separate and identical to production environment.
19. Are there specific requirements for the input data?
Data validation (credit cards, zip codes, etc.)
Data uniqueness (you cannot enter the same data more than once).
Is it time sensitive?
Business Information
20. Have you identified the performance scenarios of your system?
E.g. for Facebook application performance scenarios can be Login, viewing posts, adding posts, commenting, image uploading, sending invitations, chatting, logout etc.
21. Do you have statistics, how many users visit your site in 24 hours?
E.g. Facebook is access by more than 175 million users daily
22. How do you see your users accessing the application?
Sporadically throughout the day?
Do they all log in at simultaneously?
Certain number at specified intervals?
23. What is the average user session time on your system? E.g. Facebook user average session time is 23 minutes
24. How many transactions does the end user do per day in the application?
25. How much important is the sitting location of your end users? Do they sit in the same location or they distributed across the globe?
26. How do they access the application? E.g. RDP or Web or mobile?
27. Do you have set acceptable maximum transaction completion time?
E.g. System response time should not exceed 5 seconds while retrieving user’s order history
28. What is the peak load time on production server?
E.g. Maximum number of US based users log on to facebook.com at 8pm EST
29. On peak load time, roughly how many users are accessing the system
E.g. Facebook is accessed by up to 10 million users during peak hours
30. What will be workload model?
E.g. for Facebook users 1 million users can concurrently login, 4 million can view posts, 1 million can add posts etc.
31. How many simultaneous users, the system intends to support (please specify the number)?
E.g. Currently Facebook is supporting 10 million simultaneous users but in future it should support 20 million simultaneous users
32. Are there any time constraints for running the test?
Eg. the server can only be accessed outside business hours; server can only be accesses from 7 pm – 8 am
33. Would you like to share any additional information?
If the client didn't know anything about application. We've three ways to conduct performance testing.
1. Application in production ===> Get the details using some analytical tools.
2. Having competitors ===> Get the details of competitor's details using alexa.com etc.
3. No competitors and Application not in Production ===> Must Understand the CBT of application.
=================================================
Performance Test Scenarios Selection
Different activities (requirement gathering, scenario selection, scripting, workload model, test execution, analysis and reporting) are involved in performance testing and perfect execution of all of these is mandatory to achieve desired test results. As performance testing is a complex and time-taking activity, you can’t test each and every application scenario (unlike functional testing) in it. You select most important scenarios only, which can highlight more performance bottlenecks in AUT. It’s basically works similar to Pareto’s principle which states 80% of the issues are due to 20% of causes.
Scenario New
Selecting any additional scenario or missing out the one can greatly affect your test effort and results. Proper planning and consensus from all stakeholders is required before formally start working on AUT scenarios scripting.
Following is the list of scenarios which you should include in your performance test,
Most Frequently Accessed Scenarios: Application scenarios which are mostly accessed by end users. As such scenarios will affect maximum users they must be included in load test. For Eg, browsing product catalog in an E-commerce web application.
Business Critical Scenarios: Application scenarios where application core features exists. For Eg, purchasing a product is a business critical scenario in an E-commerce application.
Resource Intensive Scenarios: Such scenarios which are expected to consume more system resources as compared to others. For Eg, order placement will be most resource intensive scenario in an E-commerce web application.
Contractually Obligated Scenarios: Application scenarios for which company has contracted to provide hassle free services. These scenarios might not be used very frequently but they can create huge business loss in case of failure. For Eg, company claims its home page loads within 3 seconds.
Stakeholders Concerning Scenarios: Stakeholders could be more concerned on AUT new features impact on its overall performance.
Time Dependent Frequently Accessed Scenarios: Time dependent scenarios which are executed very frequently but on certain occasions only. For Eg, viewing monthly pay roll slip on an online payroll application.
Technology Specific Scenarios: These are the scenarios which are specific to AUT selected technology. For Eg, uploading a file through FTP server could be an example of technology specific scenario.
No comments:
Post a Comment