Java unit testing: how to measure memory footprint for method call

Question

Assuming I have a class that does some heavy processing, operating with several collections. What I want to do is to make sure that such operation can\'t lead to out-of-memo

Answer 1

Here is an example from Netty which does something similar: MemoryAwareThreadPoolExecutor. Guava's cache class has also a size based eviction. You could look at these sources and copy what they are doing. In particular, Here is how Netty is estimating object sizes. In essence, you'd estimate the size of the objects you generate in the method and keep a count.

Getting overall memory information (like how much heap is available/used) will help you decide how much memory usage to allocate to the method, but not to track how much memory was used by the individual method calls.

Having said that, it's very rare that you legitimately need this. In most cases, capping the memory usage by limiting how many objects can be there at a given point (e.g. by using a bounded queue) is good enough and is much, much simpler to implement.

Answer 2

You can use profiler (for ex. JProfiler) for view memory usage by classes. Or , how mentioned Areo, just print memory usage:

    Runtime runtime = Runtime.getRuntime();
    long usedMemoryBefore = runtime.totalMemory() - runtime.freeMemory();
    System.out.println("Used Memory before" + usedMemoryBefore);
        // working code here
    long usedMemoryAfter = runtime.totalMemory() - runtime.freeMemory();
    System.out.println("Memory increased:" + (usedMemoryAfter-usedMemoryBefore));

Answer 3

I can think of several options:

• Finding out how much memory your method requires via a microbenchmark (i.e. jmh).
• Building allocation strategies based on heuristic estimation. There are several open source solutions implementing class size estimation i.e. ClassSize. A much easier way could be utilizing a cache which frees rarely used objects (i.e. Guava's Cache). As mentioned by @EnnoShioji, Guava's cache has memory-based eviction policies.

You can also write your own benchmark test which counts memory. The idea is to

1. Have a single thread running.
2. Create a new array to store your objects to allocate. So these objects won't be collected during GC run.
3. System.gc(), memoryBefore = runtime.totalMemory() - runtime.freeMemory()
4. Allocate your objects. Put them into the array.
5. System.gc(), memoryAfter = runtime.totalMemory() - runtime.freeMemory()

This is a technique I used in my lightweight micro-benchmark tool which is capable of measuring memory allocation with byte-precision.

Answer 4

This question is a bit tricky, due to the way in which Java can allocate a lot of short-lived objects during processing, which will subsequently be collected during garbage collection. In the accepted answer, we cannot say with any certainty that garbage collection has been run at any given time. Even if we introduce a loop structure, with multiple System.gc() calls, garbage collection might run in between our method calls.

A better way is to instead use some variation of what is suggested in https://cruftex.net/2017/03/28/The-6-Memory-Metrics-You-Should-Track-in-Your-Java-Benchmarks.html, where System.gc() is triggered but we also wait for the reported GC count to increase:

long getGcCount() {
    long sum = 0;
    for (GarbageCollectorMXBean b : ManagementFactory.getGarbageCollectorMXBeans()) {
        long count = b.getCollectionCount();
        if (count != -1) { sum += count; }
    }
    return sum;
}

long getReallyUsedMemory() {
    long before = getGcCount();
    System.gc();
    while (getGcCount() == before);
    return getCurrentlyAllocatedMemory();
}

long getCurrentlyAllocatedMemory() {
    final Runtime runtime = Runtime.getRuntime();
    return (runtime.totalMemory() - runtime.freeMemory()) / (1024 * 1024);
}

This still gives only an approximation of the memory actually allocated by your code at a given time, but the value is typically much closer to what one would usually be interested in.

Answer 5

To measure current memory usage use :

Runtime.getRuntime().freeMemory() , Runtime.getRuntime().totalMemory()

Here is a good example: get OS-level system information

But this measurement is not precise but it can give you much information. Another problem is with GC which is unpredictable.

Answer 6

Here is a sample code to run memory usage in a separate thread. Since the GC can be triggered anytime when the process is running, this will record memory usage every second and report out the maximum memory used.

The runnable is the actual process that needs measuring, and runTimeSecs is the expected time the process will run. This is to ensure the thread calculating memory does not terminate before the actual process.

public void recordMemoryUsage(Runnable runnable, int runTimeSecs) {
    try {
        CompletableFuture<Void> mainProcessFuture = CompletableFuture.runAsync(runnable);
        CompletableFuture<Void> memUsageFuture = CompletableFuture.runAsync(() -> {


            long mem = 0;
            for (int cnt = 0; cnt < runTimeSecs; cnt++) {
                long memUsed = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
                mem = memUsed > mem ? memUsed : mem;
                try {
                    TimeUnit.SECONDS.sleep(1);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
            ;
            System.out.println("Max memory used (gb): " + mem/1000000000D);
        });

        CompletableFuture<Void> allOf = CompletableFuture.allOf(mainProcessFuture, memUsageFuture);
        allOf.get();
    } catch (Exception e) {
        e.printStackTrace();
    }
}