Baseline And Benchmark Testing
Baseline testing is the process of running a set of tests to capture performance information. This information can be used as a point of reference when in future changes are made to the application
Example: We can run baseline test of an application, collect and analyze results, and then modify several indexes on a SQL Server database and run the same test again, using the previous results to determine whether or not the new results were better, worse, or about the same.
Benchmarking is the process of comparing your system performance against an industry standard that is given by some other organization.
Performance Testing And Performance Engineering
1. Performance Testing is a distinctive QA process that occurs once a round of development is completed whereas performance engineering is an ongoing process that occurs through all phases of the development cycle i.e. from the design phase to development, to QA.
2. A dedicated performance tester or team conduct the Performance Testing who has sound knowledge of performance testing concept, tool operation, result analysis etc. A Performance Engineer is a person who has enough knowledge of application design, architecture, development, tuning, performance optimization and bottleneck root cause investigation and fixing.
3. When a bottleneck is identified during performance testing then the role of the performance tester is to analyse the test result and raise a defect. On the other hand, the job of a performance engineer is to investigate the root cause and propose the solution to resolve the bottleneck.
4. Performance Tester does not much care about the design and architecture of the application. He just focused on the application behaviour under load whereas Performance Engineer cares about how efficient each component of the application is performing under load.
5. Performance Testing Life Cycle (PTLC) covers all phases of Performance Testing whereas Performance Engineering Life Cycle (PELC) covers all the engineering activities and deliverables.
Difference between Performance center and Controller ?
Performance center
1. PC is web interface of the controller
2. we can book the slot to execute the test.
3. we can download the scripts and also .lrr files where ever we are.
Controller
1. controller is stand alone machine.
2. we can not book the slot to execute the test.
3. Not possible to download .lrr file.
Difference between Web Server and Application Server ?
- Web servers are desirable for static content whereas application servers are appropriate for the dynamic content.
- Web servers support scripting languages like Perl, PHP, ASP, JSP, etc. As against, application server assists scripting languages as well as application level services such as connection pooling, transaction support, object pooling, messaging services etc.
- Application server contains web and EJB containers and a web server as an incorporated part of them. In contrast, a web server only contains web or servlet container and can employ EJB.
- Web server does not support multithreading whereas application server assists multithreading and distributed transaction.
- Web server uses HTML and HTTP protocol. On the other hand, the application server could use graphical user interface and protocols like RPC/RMI including HTTP.
- Load Limit or capacity is higher in case of the application server as compared to the web server.
- The web server provides an environment to run a web application and features like caching and scalability. On the contrary, the application server provides an environment to run web with enterprise applications.
Differences between Request and Hit?
Request : User action is a request.
Hit: Successful request is a hit.
web_submit_data and web_submit_form
Ans : web_submit_data : Specific to the page and context less
web_submit_form : Specific to the form and context base
context base means the request will be dependent on the previous response.
Ans : web_submit_data : Specific to the page and context less
web_submit_form : Specific to the form and context base
context base means the request will be dependent on the previous response.
lr_eval_string and lr_save_string
lr_eval_string : It reads value from the variable
Replace all occurrences of a parameter with its current value.
To save the parameter value to other parameter using lr_eval_string
lr_save_string : it assign the value to lr variable.
it save the null terminated string to a parameter
syntax : lr_save_string("cstring", "LR string");
eg : lr_save_string("some string value", "prm_str");
lr_output_message(lr_eval_string("value of stri :", {prm_str}));
To save the parameter value to other parameter using lr_eval_string
lr_save_string : it assign the value to lr variable.
it save the null terminated string to a parameter
syntax : lr_save_string("cstring", "LR string");
eg : lr_save_string("some string value", "prm_str");
lr_output_message(lr_eval_string("value of stri :", {prm_str}));
difference between lr_error_message () and lr_output_message()?
lr_error_message () function we can send the error message to the controller message area but using lr_output_message() we didn't.
SOAP and REST
SOAP
- SOAP is a protocol
- SOAP is Simple Object Access Protocol
- SOAP is supports WS-security and SSL
- SOAP requires more resources and bandwidth
- SOAP only works with XML formats
- SOAP uses service interface to expose it functionality to client application.
REST :
- Rest is Architectural Pattern
- Rest stands for Representational State Transfer Protocol
- REST works with plain text, XML. HTML & Json.
- REST does not need much bandwidth
- REST uses uniform service locators to access to the components on the hardware device
Difference between GET and POST
GET :
- GET is not a secured request
- It will send small amount of information to the server
- It can be available in history cached and bookmarks
- GET request is often cacheable
- To retrieve the information
- This method supports only string data types.
POST :
- POST is a secured request
- It will send large amount of information
- It can't be available in history, cached and bookmarks
- POST request is NOT cacheable
- To submit the data to server
- This method supports different data types, such as string, numeric, binary, etc.
Difference between HTML mode and URL mode
Context Dependency: HTML mode performs context-based recording whereas URL mode is free from context i.e. contextless recording. Since the resources are separated by individual request in URL mode, so there is no context dependency between them.
Resources: In HTML mode, the page resources like image, .css etc. are recorded in a single request whereas individual requests are created for each resource in URL mode.
User Action: In HTML mode, each user action belongs to one request whereas in URL mode, multiple requests associated with one user action depends on the number of resources available on the page.
Script Size: The size of the HTML mode script is comparatively smaller than the size of the URL mode script. The reason is in URL mode script each resource has a separate request.
Correlation Effort: HTML mode script requires less effort than URL mode script to correlate the dynamic value.
Resource Download during replay: Resources are downloaded during replay in HTML mode. But in URL mode individual request triggers for each resource hence resource downloading is not required.
Execution Speed: HTML mode script is slower than URL mode script due to parsing of code.
Script Maintenance: The maintenance of the script is less in HTML mode. On the other hand, the URL mode script requires more maintenance and re-work due to changes in the application.
Explicit support: HTML mode supports explicit URL whereas URL mode does not support explicit HTML.
Used for: HTML mode is used for browser-based application whereas URL mode is used for non-browser (thick-client) application.
Functions: In HTML mode, web_submit_form is used whereas in URL mode, web_submit_data, web_image etc. are used.
Difference between WinInet and Socket Level
Sockets-based (default)
WinInet based.
The WinInet is the engine used by Internet Explorer and it supports all of the features incorporated into the IE browser. The limitations of the WinInet replay engine are that it is not scalable and does not support Linux. In addition, when working with threads, the WinInet engine does not accurately emulate the modem speed and number of connections.
VuGen’s proprietary sockets-based replay is a lighter engine that is scalable for load testing. It is also accurate when working with threads. The limitation of the sockets-based engine is that it does not support SOCKS proxy. If you are recording in that type of environment, use the WinInet replay engine.
Difference between Entry Criteria and Exit Criteria :
Entry criteria are:
- Finalized NFRs
- Script completion
- Test environment must be ready
- Deployment of the latest functionally tested code
- Test Data readiness
Exit criteria are:
- NFRs must be met
- No performance bottleneck
- No open defect
- Final performance test report submission
Run Vuser as a Thread” vs “Run Vuser as a Process
. What Is The Advantage Of Running The Vuser As Thread?
Answer : VuGen provides the facility to use multi-threading. This enables more Vusers to be run per generator. If the Vuser is run as a process, the same driver program is loaded into memory for each Vuser, thus taking up a large amount of memory. This limits the number of Vusers that can be run on a single generator.
If the Vuser is run as a thread, only one instance of the driver program is loaded into memory for the given number of Vusers (say 100). Each thread shares the memory of the parent driver program, thus enabling more Vusers to be run per generator.
A generic statement is Run Vuser as a thread is used for web-based applications whereas
Run Vuser as a process is used for client-based applications testing.
This feature gives an option to enable the multithreading mode of the Vusers as per requirement. To enable it you can go to:
VuGen -> Runtime Setting -> General -> Miscellaneous -> Multithreading
Run as a Thread vs Run as a Process
LoadRunner Runtime Settings, you will find 2 radio buttons “Run Vuser as a thread” and “Run Vuser as a process”. You need to select one of them for your test. What are the criteria of selection ?
As per Java Multithreading definition, both processes and threads are independent sequences of execution, so running a Vuser as a thread or as a process is an independent task for each Vuser.
there are some major differences between both the options which are covered below:
Run Vuser as a thread vs Run Vuser as a process:
By Definition: A program (i.e. LoadRunner script) is referred to as a process while a thread is a subset of the process, also referred to as a lightweight process.
Program Loading:
In Run Vuser as a process mode, the controller loads the same driver program into the memory again and again for every instance of the Vuser and increases the need for RAM.
In Run Vuser as a thread mode, the controller loads only one instance of the driver program for all the threads.
Memory Sharing:
Running Vuser as a process mode needs separate memory space for each thread which is not shared with any other thread.
In Running Vuser as a thread option, the allocated memory space is shared between all the threads.
Memory Consumption:
Running Vusers as a thread mode requires less memory due to its memory sharing feature.
Running Vusers as a process mode needs significantly more memory space.
Vuser Count:
Running Vuser as a thread mode can accommodate more Vusers count than Running Vuser as a process mode.
Execution:
Running Vuser as a process runs separately using allocated resources. There should be at least one thread of that process called primary thread (multithreading basic).
Running Vuser as a thread creates multiple threads of a single process. These threads execute concurrently and share the allocated resources of the process.
Number of Processes:
Running Vusers as a thread initiates only 1 process i.e. mdrv.exe for all the threads. On the other hand, Running Vusers as a process launches individual process for each thread.
Example: If 10 Vusers are running as a thread then there will be only 1 mdrv.exe process in the task manager whereas you can see 10 different mdrv.exe processes while running the test with Run Vuser as a process mode.
Killing mdrv.exe:
On killing the mdrv.exe process, all the threads will stop and exit the test in Run Vuser as threads mode whereas in Run Vuser as a process, only associated thread will stop and exit the test on killing a particular mdrv.exe process.
Where to use:
A generic statement is Run Vuser as a thread is used for web-based applications whereas Run Vuser as a process is used for client-based applications testing.
In case of any unhandled exception (in LR code, not the script code) caused by one of the Vusers which in turn killing the .mdrv process that leads failing of all the running Vusers. In such case Running Vuser as a process mode will be the best option because termination of one .mdrv process does not interrupt the functioning of other processes.
One Key Point:
If the application supports multithreading feature then the Vuser script will also support multithreading and you can choose Run Vuser as a thread mode.
Additional Information on Process and Thread:
Attributes:
Virtual address space, global variables, open files, child processes, executable code, a security context, a unique process identifier, a priority class, minimum and maximum working set sizes, and at least one thread of execution all these are associated with Running Vuser as a process.
Program Counters, Registers, Stack, Thread State, scheduling priority, thread local storage, a unique thread identifier all these are associated with Running Vuser as a thread.
Communication:
A thread uses methods like wait(), notify(), notifyAll() to communicate with other thread (of the same process). A process can communicate with other processes by using inter-process communication only.
Starting of New instance:
The creation of the thread is easy and requires less time. However, the creation of a new process requires the duplication of the parent process and need more time than a thread creation.
Control:
Threads have control over the other threads of the same process. A process does not have control over the sibling process, it has control over its child processes only.
Context Switching:
Thread context switching is faster than process context switching.
Creation and Termination:
Thread creation and termination are quicker than process creation and termination.
=========================
Hits/sec and Transaction/sec :-
A Transaction is a group of requests which creates multiple Hits on the server. Hence we will see multiple Hits against one Transaction.
Single Transaction can create multiple Hits on the server.
Hits/Sec graph helps to identify the request rate sent by the testing tool.
High Response Time may cause less Hits/Sec
Eg : A Login operation which involves 10 HTTP requests can be group together into a single transaction.
when you see 1 dot in TXN/Sec graph there will be a 10 dots in Hit/sec Graph.
Absolute Graph and Relative Graph :
Absolute Graph : Based on the System Time
Relative Graph : Based on the Elapsed Time
Simultaneous Users and Concurrent Users :
Difference Between Overlay Graph And Correlate Graph?
Overlay Graph: It overlay the content of two graphs that shares a common x-axis. Left Y-axis on the merged graph shows the current graphs value & Right Y-axis show the value of Y-axis of the graph that was merged.
Correlate Graph: Plot the Y-axis of two graphs against each other. The active graphs Y-axis becomes X-axis of merged graph. Y-axis of the graph that was merged becomes merged graphs Y-axis.
Difference between values of Average response time in Summary Report and Average transaction response time graph
Sometimes it is confusing, when we observe the difference in Avg, max and min values of response time in “Summary report” and values in “Average transaction response time” graph.
To understand this we will have to understand how these values are calculated.
The Avg, max and min values of response time in “Summary report” is calculated using the complete data of transactions that executed during the test duration and is most precise.
The “Average transaction response time” graph does not use all the data points captured during execution. The values in the Graph are averaged out based on the granularity to make it more readable. So, the the Avg, max and min values of response time in “Average transaction response time” graph is calculated based on the averaged values (calculated based on granularity) used as data points to plot this graph. These values will change if there is change in the granularity.
You may also find that the Avg, max and min values of response time shown during execution in controller is different from the above two. It could be because there is some think time coded with in the transaction scope and it is added to the response time value, shown in controller. This think time is excluded by default in the LoadRunner analysis.
Difference between Summary Data and Complete Data ?
Summary Data :Raw & Unprocessed Data
Complete Data : It refers the Result Data after it has been Processed for use with analysis.
Difference between Summary Report and Transaction Analysis Report ?
Summary Report : It provides the General information about the scenario run.
You can access the Summary Report at any time from the Session Explorer.
Transaction Analysis Report :- It provides a detailed analysis of specific Transaction over a specific Time period.
===
Difference Between Performance Testing And Functional Testing?
Functional Testing : To verify the accuracy of the software with definite inputs against expected output, functional testing is done
This testing can be done manually or automated
One user performs all the operations
Customer, Tester and Development involvement is required
Production sized test environment is not necessary, and H/W requirements are minimal.
Performance Testing : To validate the behavior of the system at various load conditions performance testing is done.
It gives the best result if automated
Several user performs desired operations
Customer, Tester, Developer, DBA and N/W management team
Requires close to production test environment and several H/W facilities to populate the load
Difference between Page Load Time and Page Rendering Time ?
Page Load Time :- How much time it is taking the page to load all the content in the page.
Page Rendering Time : Total time is taking to get the immediate next page.
Difference between Cookie, Cache and Session ?
– Cookies keep information such as user preferences, while
Cache will keep resource files such as audio, video or flash files.
– Typically, Cookies expire after some time, but cache is kept in the client’s machine until they are removed manually by the user.
Cookies are client-side files that contain user information, whereas Sessions are server-side files that contain user information.
Cookie is not dependent on session, but Session is dependent on Cookie.
Cookie expires depending on the lifetime you set for it, while a Session ends when a user closes his/her browser.
The maximum cookie size is 4KB whereas in session, you can store as much data as you like.
Cookie does not have a function named unsetcookie() while in Session you can use Session_destroy(); which is used to destroy all registered data or to unset some
Difference between Soft Parse and Hard Parse ? which one is best ?
Soft Parse is good.
Oracle SQL is parsed, before execution checked for syntax (and parts of the semantic check) the SQL is loaded into the library cache.
soft parse does not require a shared pool reload (and the associated RAM memory allocation).
Oracle should be perform a syntax check and semantic check because it is possible that a DDL change altered one of the target tables or views
Whenever we run a SQL qry, the soft parse is directly taken from Buffer cache or Library Cache and it will executed very fastly.
A hard parse is when your SQL must be re-loaded into the shared pool.
A hard parse is worse than a soft parse because of the overhead involved in shared pool RAM allocation and memory management.
A Hard Parse is taken more and more time to execute SQl qry.
Difference between Web_add_header And Web_add_auto_header ?
Web_add_header: Web_add_header function is used to add header only to HTTP request that follows it.
Example:
web_add_header("Cookie", "{Cookie_ID}");
web_add_header("Pragma", "no-cache");
web_url("webmail",
"URL=http://{URL}/etc/apps/webmail/",
"Resource=0",
"RecContentType=text/html",
"Referer=",
"Snapshot=t1.inf",
"Mode=HTML",
LAST);
web_add_header("Cookie", "{Cookie_ID}");
web_add_header("Pragma", "no-cache");
web_url("webmail_3",
"URL=http://{URL}/etc/apps/webmail/?_task=mail&_action=getunread&_remote=1&_unlock=0&_={TimeStamp}",
"Resource=0",
"RecContentType=text/plain",
"Referer=http://{URL}/etc/apps/webmail/?_task=mail",
"Snapshot=t2.inf",
"Mode=HTML",
LAST);
web_add_auto_header: Web_add_auto_header function is used to add header to all the consecutive HTTP requests.
Web scripts usually send the standard header requests automatically for each request. If you need additional headers to be sent then you can use web_add_header or web_add_auto_header.
web_add_header only sends to the HTTP request that follows it, whereas web_add_auto_header sends to all the succeeding requests.
Web_add_header and web_add_auto_header is automatically generated in your script, if you enable this in Record -> recording options->Advanced->Headers->Record Headers not in list.
Example:
web_add_auto_header("Cookie", "{Cookie_ID}");
web_add_auto_header("Pragma", "no-cache");
web_url("webmail",
"URL=http://{URL}/etc/apps/webmail/",
"Resource=0",
"RecContentType=text/html",
"Referer=",
"Snapshot=t1.inf",
"Mode=HTML",
LAST);
web_url("webmail_3",
"URL=http://{URL}/etc/apps/webmail/?_task=mail&_action=getunread&_remote=1&_unlock=0&_={TimeStamp}",
"Resource=0",
"RecContentType=text/plain",
"Referer=http://{URL}/etc/apps/webmail/?_task=mail",
"Snapshot=t3.inf",
"Mode=HTML",
LAST);
