JVM is a Dynamic Memory it is stored in RAM.
Defi :- After using the Object memory, it should get free. If it is not getting free then that is called MemoryLeak.
-> To Avoid
OutOfMemory Exception which causing Memory Leak to do Increase the
HEAP size from
512 MB to 1024 MB or else to increase from ...... MB to ......MB.
-> If there is no Memory space present for creating new objects in HEAP, JAVA throws OutOfMemory Exception or Java.Lang.OutOfMemory Error.
Q : What will happen if HEAP size is low ?
Ans : The Number of GC Cycles are increase and the GC frequency will come very often.
every time it will create the "STOP the Events"
It will literally halt the applications.
It will suspend and terminates all your threads.
Heap is the space where the Objects & References are stored.
All the Object related data is stored in HEAP Area.
(Object related Data means Object Information, Object RunTime Data and all the Instance variables Information. This can be accessed by Multiple Memory).
Each JVM has its own memory segment, that is called HEAP Memory, which is storage of Java objects.
These Java objects can be grouped based on their age, like
- Younger Generation ,
- Older Generation and
- Permanent Generation.
Note :- Every JVM having only one Heap Area
Heap and GC are decide the performance of JVM.
GC is a Mechanism which will have some Automatic Memory Management.
It is having own unique GC Memory Algorithm which are used remove unreachable objects or Unreferenced Objects from Memory.
Live Objects are Reachable
Dead Objects are Not Reachable
Some Important Terms :
- Live Objects
- Dead Objects
- Demon Thread
- System.GC( );
Daemon Thread : Created by JVM
Non-Daemon Thread : Created by Application
Note : -The Main Thread is always Non-Demon Thread.
Native Thread : 1. Threads in this model are managed by the JVM with the help of underlying OS support.
2. These threads are implemented at the OS level and managed in kernel space.
GC is process of automatically freezing objects that are no longer referenced by the program.
Each Generation has its own memory segment with in the heap.
When this segment is full, GC deletes all the unreferenced objects that are moved as Garbage to create space.
If the HEAP size is decreased gradually overtime then its clear that there is a some problem.
If you are running load test for a long time and you observe that the heap size is getting decreased, then one of the problem could be Memory Leak.
Generally an object becomes eligible for garbage collection in Java on following cases:
1) All references of that object explicitly set to null e.g. object = null
2) Object is created inside a block and reference goes out scope once control exit that block.
3) Parent object set to null, if an object holds reference of another object and when you set container object’s reference null, child or contained object automatically becomes eligible for garbage collection.
4) If an object has only live references via WeakHashMap it will be eligible for garbage collection.
Note :- The HEAP Memory is different from Actual Memory.
We can identify the Memory leaks by running a load test an application for a prolonged duration of 10 to 12 hours and monitoring the memory utilization.
Note : -
If the Application happens to have Memory Leak then FullGC will start to run repeatedly without reclaiming the Memory.
When full GC runs repeatedly CPU will start to spike up and never comes down.
Sol :- To resolve the problem, Memory leak has to be fixed using Thread Dump/
Heap Dump root cause of the problem should be identified & Fixed.
Reasons for CPU Spike up : -
1. Repeated full GC
2. Non Terminated Loops
3. Non Synchronized to Java.Util.HashMap.
GC : Collecting unused objects
If Younger generation is filled and not able to free the memory to create New Objects, then it is completely Developer Issue.
High GC can Lead High CPU Utilization and slow Response Time. This is due to suspensions.
GC Mechanism :
Half GC :- Deallocate the Memory from Younger Generation
Full GC :- Deallocate the Memory from Both Generations
Heap and GC are decide the Performance of JVM.
Syntax : jmap -dump:[live],format=b,file=<file-path> <pid>
Eg : jmap -dump:format=b,file=/opt/tmp/heapdump.bin 37320
How do you find JVM free Memory ?
Ans : XMX, XMS
java -XX:+HeadDumpOnOutOfMemoryError -XX:HeapDumpPath=<file-or-dir-path>
When memory usage exceeds beyond XMX we will get JVM out of memory Exception.
To check Memory Leaks : System Resource Graph -> Windows Resource Graph.
Thread do not Occupy Memory space with in Java Heap
They Occupy Memory space outside the Java Heap
This is one of the reason why typically java process taking more than XMX value
Memory Leak Symptoms :
Performance symptoms during memory Leakage in our Application :
-> The Application uses Heap memory increased by time
-> The Response slowdown gradually due to memory cognition
-> Aggressive execution of GC activity.
-> HEAP Dump shows a lot of objects retained(from the leakage types)
-> A Sudden increase of memory PAGING as reported by the OS monitoring tools.
-> OUT OF MEMORY occurs frequently in the logs and some times an application server restart its required.
Due to Memory Leak affects:
-> Slow system performance.
-> Crashes in running programs.
-> The System may hang
-> The System reboot all the resources including the Leaked Memory.
Main Causes of OUTOFMEMORY Exception:
-> It is always Memory Leak.
-> GC is not freezing up the memory because there can be a dead locks.
-> Memory Fragmentations.
-> Excess GC Overhead
-> Allocating oversized Temp Objects.
-> Non Terminated Loops
-> Repeated full GC
Note : If Application creates too many Threads then it can be result in "Java.Lang.OutOfMemory" Error: Unable to create New Native Thread".
NOTE : when you identify
High Response time along with
MEMORY Leak, so first carryout
HEAP Dump analysis and check the frequency of GC and carryout the GC analysis to identify the root cause analysis.
NOTE : If the HEAP size is decreased gradually overtime, then it is clear that is a server problem.
if you are running a Load Test for a long time and you deserve that the HEAP size is getting decreased, then there is one of the problem could be a MEMORY Leak.
NOTE : If Total Memory usage is increasing but Logical Thread and Private Byte counters are not increasing, Then there is a Leak in MEMORY.
HEAP :
JVM created HEAP memory for allocated objects.
It is a snapshot of a memory which contains Java Objects and class information.
Q : can we change the HEAP settings during load testing ?
Ans : Yes, We can, but it will effect the Performance issue.
If you change the HEAP setting during Load testing, you need to restart the JVM.
Q : Where we get OutOfMemory Exception in GC ?
Ans : When Memory usage exceeds beyond XMX we get JVM OutOfMemory Exception.
The Min HEAP size is XMS
The Max HEAP size is XMX
XMN :- set the initial and MAX size of the Heap for the Young Generation.
Q : Where we need to increase the Heap size ?
Ans : In Tomcat Catelina file
In WebLogic set DomainEnv.cmd file
set WLS_MEM_ARGS_64bit = -XMS 512M -XMX 1024 M
set WLS_MEM_ARGS_32bit = -XMS 512 M -XMX 512M
Note : Serial GC will work only Standalone applications and Desktop Applications.
If we use distributed application we will get High Performance issues.
CPU :-
IF CPU is a Bottleneck :
Add a Longer Pacing Time between Iterations
Add Wait steps of a second or two
IF Memory is a Bottleneck run fewer VUsers
High CPU Utilization occur whenever excessive GC cycles Running
High GC can lead to High CPU Utilization and slow Response time this is due to Suspensions.
Note :- To check Memory Leaks = System Resource Graph -> Windows Resource Graph.
Flags :-
To change Type of GC's
Java -XX:+PrintCommandLineFlags -version
Java -XX:+UseG1GC -XX:+PrintCommandLineFlags -Version
Java -XX:+UseSerialGC -XX:+PrintCommandLineFlags -Version
Memory Increases below two Reasons :
A Bad GC Memory settings or an Application Problem
The Application Problems is either a MemoryLeak or one of the several other memory Problems.
- whenever available Memory goes down
- Some sudden failures of Application
eg : sudden Response Time Increase
High CPU Utilization (it is occur whenever excessive GC cycles Running)
In 100% Memory 40% Filled
- Extremely poor Response Time.
- Hung Application
- Very Slow startup
- Unusual amount of Resource Utilization.
Note : Before generating Thread Dump we must know Java Process ID,
We can get that ID by using JPS Command in UNIX Server.
eg : $JPS
Note - Below Performance symptoms that usually appear incase of Thread Blocking.
1. Slow Application Response.
2. App Server logs might be show some stuck Threads
3. The Servers healthy status becomes critical on Monitoring Tools.
4. Frequent App Server restarts manually or Automatically.
5. Thread Dump shows a lot of Threads in the Blocked status waiting for different resources.
6. App Profiling shows a lot of Thread blocking.
What is Memory leak:
when ever a computer program consumes memory but unable to release it back to the operating system. A memory leak has symptoms similar to a number of other problems and generally can only be diagnosed by a programmer with access to the program source code; however, many people refer to any unwanted increase in memory usage as a memory leak, though this is not strictly accurate.
The memory for any consistent increase and also any degradation in CPU performance. Is it a memory leak?
Note that constantly increasing memory usage is not necessarily evidence of a memory leak.
Some applications will store ever increasing amounts of information in memory (e.g. as a cache). If the cache can grow so large as to cause problems, this may be a programming or design error, but is not a memory leak as the information remains nominally in use.
In other cases, programs may require an unreasonably large amount of memory because the programmer has assumed memory is always sufficient for a particular task; for example, a graphics file processor might start by reading the entire contents of an image file and storing it all into memory, something that is not viable where a very large image exceeds available memory.
To put it another way, a memory leak arises from a particular kind of programming error, and without access to the program code, someone seeing symptoms can only guess that there might be a memory leak. It would be better to use terms such as "constantly increasing memory use" where no such inside knowledge exists.
The term "memory leak" is evocative and non-programmers especially can become so attached to the term as to use it for completely unrelated memory issues such as buffer overrun.
Checking for Leaks: There are a number of telltale signs that an application is leaking memory.
- Maybe it's throwing an OutOfMemoryException.
- Maybe its responsiveness is growing very sluggish because it started swapping virtual memory to disk.
- Maybe memory use is gradually (or not so gradually) increasing in Task Manager.
When a memory leak is suspected, you must first determine what kind of memory is leaking, as that will allow you to focus your debugging efforts in the correct area.
Use PerfMon to examine the following performance counters for the application:
Process/Private Bytes:
The Process/Private Bytes counter reports all memory that is exclusively allocated for a process and can't be shared with other processes on the system.
Test: If Process/Private Bytes is increasing, but # Bytes in All Heaps remains stable, un managed memory is leaking.
.NET CLR LocksAndThreads/# of current logical Threads:
The .NET CLR LocksAndThreads/# of current logical Threads counter reports the number of logical threads in an AppDomain.
Test: If an application's logical thread count is increasing unexpectedly, thread stacks are leaking.
Test: If both counters for 'logical thread count' and 'Private Bytes' are increasing, memory in the managed heaps is building up.
.NET CLR Memory/# Bytes in All Heaps:
The .NET CLR Memory/# Bytes in All Heaps counter reports the combined total size of the Gen0, Gen1, Gen2, and large object heaps.
Test:
By default, the stack size on modern desktop and server versions of Windows? is 1MB. So if an application's Process/Private Bytes is periodically jumping in 1MB increments with a corresponding increase in .NET CLR LocksAndThreads/# of current logical Threads, a thread stack leak is very likely the culprit.
Test:
If total memory use is increasing, but counters for 'logical thread count' and 'Private Bytes' (measuring managed heap memory) are not increasing, there is a leak in the unmanage
Alternative method:
Start with monitoring the response times, throughput, total tps etc.. You should see the impact here if not monitoring the run time environment or system resources in first instance. Now it could or could not be memory leak.
Look at memory profile of the server hosting the run time environment and application server logs. Check the logs, if out of memory errors are recorded in the logs it could or could not be a memory leak. Check heap usage and gc logs. It could be a memory leak if the heap is full and no memory is being released after gc(s). If there is enough heap but jvm is still kicking off gc's to free the memory, the perm gen space might be full or could be some other reason.
If its a memor leak then jvm would be thrashing and hogging up all the cpu. You won't see any load on the down stream systems. Plotting graph from gc logs would show an increase in the heap troughs.
Above is just one example and there could be many many variations to this. You can simulate a memory leak yourself, just google it and you will find code to both induce and fix it.
As you might guess, memory leak, if left unattended and not corrected, could prove to be fatal. Memory leaks can be found out by running tests for long duration (say about an hour) and continuously checking memory usage.
Issues caused by memory leaks are essentially based on two variables for a standalone windows application
1) Frequency of usage
2) Size of memory leak.
If either one or both are very high, the computer might come to a point when no memory is available for other applications. This could lead to a computer crash. If it is a network based application then you will also have to consider network traffic. If each network transaction causes a memory leak, then a high volume of network transactions could also prove dangerous.
Endurance Testing
Soak testing, endurance testing or stability testing; the terms are largely interchangeable but refer to a test that is designed largely around running a controlled load, usually a little lower than business-as-usual, over a prolonged period. This tests for resource consumption and non-release that can only be identified over time. Gradual memory leaks are the classic example,
where available memory might reduce little by little. In the short term this will have no effect on the system, but over time could be a major resourcing issue.
Log files tend to grow over time, and past a certain point may cause system issues if there is insufficient disk space, garbage clean up processing, etc. This small cumulative growth and non-release can all be issues that are only apparent over time.
So how long should a soak test run? As long as possible. In reality you should aim for anything over 12 hours. Ideally a soak test would run for 72 hours, so the system can demonstrate its ability to hold things together for a period longer than a weekend's-worth of non-stop operation. If the system continues to operate normally for 72 hours, it gets over the period it generally will need to be supported by out-of-hours on-call staff. This is generally the critical length of time a system needs to be able run "unattended," as weekend support is far more costly to an organization than business-hour support.
Note :
It is used for "To find out Memory leaks",
It is used for "To find out Connection failures between different Layers",
It is used for "To find out Connection Failure in DB connection"
============
if your LG exceeds the 80% CPU and you get that warning, i'd treat the results with suspicion.
the next questions are points you need to consider:
How many LGs did you use for this test
Did you get the error on all your LGs
How many Vusers did you run in this test?
At what point in your ramp up did you start getting the CPU error messages?
Once you answer all these, try the following:
Run with a lower amount of Vusers on your LG and see if the error messages still pop up, or
Run your test with more than 1 LG and distribute your Vusers accordingly between the LGs
Improve the HW on your LG
====================
As we know that memory is a holding place for instructions and data that the microprocessor can reach quickly. So while processing, the memory contains the main parts of the operating system and some or all of the application programs and related data that are currently being used.
Below diagram can give you some inference on where the data can be stored and which storage areas can render faster response.
Threads & Process
So while seeing about memory, there are two elements that comes into picture.
Process is one instance of a running application and all the memory and other resources associated with it. Threads on the other side, is one path of execution through application’s code. A process can consist of one or many threads of execution.
So what is a bottleneck?
A component is said to be bottleneck whose performance limits the optimum performance of other components. It can because of the any of the following reasons:
- Insufficient resources
- Resource malfunctioning
- Uneven workload sharing
- Incorrect configuration of resources
- Monopolization of a particular resource
Paging/Swapping
Paging is a movement of pages of data between disk and main merry. when there are too many pages in memory, the RAM shortens in capacity to have all the pages in it, so it uses 'virtual memory'. When each page in execution demands that page that is not currently in real memory (RAM), it places some pages on virtual memory and adjusts the required page on RAM. The state of excessive paging is called Thrashing. It is a state in which the CPU performs 'productive' work less and 'swapping' more. This could lead to a major memory bottleneck.
Performance Counters
Let’s see some of the memory counters now and how these counters can be added in a Windows OS.
Memory Object
The memory object describes the behaviors of physical and virtual memory on the computer
Physical memory is the amount of RAM on the computer. Virtual memory consists of space in physical memory and on disk.
How to add memory counters:
- Go to perfmon utility.
- click on Add counters
- Choose Memory as performance object
- Select the counters that you want to monitor.
Memory Cache
This is a place where the processor stores the data or instructions that is currently working at the time or is predicted to work shortly. It allows the processor to get the information quickly from the faster cache memory. The net result is a more efficient and faster running system.
Memory Leak
It occurs when applications allocate memory for use but don not free allocated memory when finished.. It causes temporary memory shortages in application programs that run for a short time. It causes system to allocate all available memory to one process. Eventually, the system hangs till the memory is released.
Tuning Tips for memory resources:
- Increase physical memory about above the minimum required.
- Create multiple paging files while using multiple disks.
- Determine the correct size for the paging files
- The initial size fi the paging file is between 1 and 1.5 times the amount of RAM available
- Check the available space on your disks. Do not use large paging files.
- Ensure effective usage of Cache memory.
- Monitor the applications and replace those that leak memory or use it inefficiently
- Shut down the services that are not required by the application
- Replace 16-bit system with 32-bit or 64-bit systems
- Run memory-intensive programs on the high end computers or when the system workload is light.
Reasons for memory bottlenecks
Too many page faults - Having too many page faults leads to excessive program execution delays. Ensure that your application doesn’t experience too many hard faults
Disk contention
Competition for memory - When memory is scarce, the memory access pattern of one program can unduly influence other running programs.
Ques : - How to graph the Partial GC and Full GC of an application in Dynatrace ?
Ans : In Dynatrace go to Agent based Measures -> JVM ->(here having)
-> Committed Memory, MAX Memory, Memory pool, Thread Count, Memory Utilization
-> Free Memory, Specific GC Activation, Total GC Activation.
-> GC Collection Old Memory & Young Generation, Un Loaded Classes, Used Memory, etc...
Decreasing Full GC Time
The execution time of Full GC is relatively longer than that of Minor GC. Therefore, if it takes too much time to execute Full GC (1 second or more), timeout may occur in several connected parts.
If you try to decrease the Old area size to decrease Full GC execution time, OutOfMemoryError may occur or the number of Full GCs may increase.
Alternatively, if you try to decrease the number of Full GC by increasing the Old area size, the execution time will be increased.
