How do I measure fragmentation in Hotspot's Metaspace?

问题

I'm looking into debugging an "OutOfMemoryError: Metaspace" error in my application. Right before the OOME I see the following in the gc logs:

{Heap before GC invocations=6104 (full 39):
 par new generation   total 943744K, used 0K [...)
  eden space 838912K,   0% used [...)
  from space 104832K,   0% used [...)
  to   space 104832K,   0% used [...)
 concurrent mark-sweep generation total 2097152K, used 624109K [...)
 Metaspace       used 352638K, capacity 487488K, committed 786432K, reserved 1775616K
  class space    used 36291K, capacity 40194K, committed 59988K, reserved 1048576K
2015-08-11T20:34:13.303+0000: 105892.129: [Full GC (Last ditch collection) 105892.129: [CMS: 624109K->623387K(2097152K), 3.4208207 secs] 624109K->623387K(3040896K), [Metaspace: 352638K->352638K(1775616K)], 3.4215100 secs] [Times: user=3.42 sys=0.00, real=3.42 secs] 
Heap after GC invocations=6105 (full 40):
 par new generation   total 943744K, used 0K [...)
  eden space 838912K,   0% used [...)
  from space 104832K,   0% used [...)
  to   space 104832K,   0% used [...)
 concurrent mark-sweep generation total 2097152K, used 623387K [...)
 Metaspace       used 352638K, capacity 487488K, committed 786432K, reserved 1775616K
  class space    used 36291K, capacity 40194K, committed 59988K, reserved 1048576K
}

From what I can see, Metaspace capacity isn't even nearing the committed size (in this case, -XX:MaxMetaspaceSize=768m). So I suspect fragmentation of Metaspace causing the allocator to fail to find a new chunk for the new classloader.

I'm aware of -XX:PrintFLSStatistics but that only covers CMS, not native memory.

So my question is: is there a debugging help similar to PrintFLSStatistics available for Hotspot's native memory?

This is using Java HotSpot(TM) 64-Bit Server VM (25.45-b02) for linux-amd64 JRE (1.8.0_45-b14).

回答1:

I've just looked into the implementation of the Metaspace in HotSpot. The Metaspace is divided into chunks and managed using a freelist. So fragmentation is indeed a possible reason for your problem.

I've also looked through the flags of the HotSpot VM (-XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal), there is no flag in the release version.

However, there is a dump() method in the Metaspace class which seems to be triggered by setting the -XX:+TraceMetadataChunkAllocation flag. There is also the -XX:+TraceMetavirtualspaceAllocation which is sounding to be of interest for you. However, those are "develop" flags, meaning you need a debug version of the VM.

回答2:

@loonytune's answer works just fine, but I want to provide a little bit more detail:

For context, "The Metaspace" is a collection of metaspaces, one per class loader. Each metaspace holds a list of VirtualSpace objects out of which Metachunks of different sizes are allocated. These chunks hold MetaBlocks, which are the real containers for metadata.

I need a debug JRE to run those flags, so following this tuorial I checked out the openjdk repository (I renamed the checkout to vm because the build scripts seem to take issue with the jdk8 folder name), ran

~/vm$ bash configure --enable-debug
~/vm$ DISABLE_HOTSPOT_OS_VERSION_CHECK=ok make all

and used the resulting vm/build/linux-x86_64-normal-server-fastdebug/images/j2re-image as my java runtime.

The log lines generated look like this:

VirtualSpaceNode::take_from_committed() not available 8192 words space @ 0x00007fee4cdb9350 128K, 94% used [0x00007fedf5e22000, 0x00007fedf5f13000, 0x00007fedf5f22000, 0x00007fedf6022000)

Which indicates that the current VirtualSpace is full and can't hold another chunk of the requested 8192 word size. This will cause this metaspace to switch to another VirtualSpace.

ChunkManager::chunk_freelist_allocate: 0x00007fee4c0c39f8 chunk 0x00007fee15397400 size 128 count 0 Free chunk total 7680 count 15

ChunkManager::chunk_freelist_allocate: 0x00007fee4c0c39f8 chunk 0x00007fedf6021000 size 512 count 14 Free chunk total 7168 count 14

This happens when a new Metachunk is allocated, in the first case it's 128 words big and uses up the list of small chunks. As you can see, the next request goes to the medium sized chunks (of size 512) and leaves 14 chunks free in total. Once the free total reaches 0, a Full GC is needed to increase the total Metaspace size.

Note that specifying -verbose gets you even more output from the above two flags.

来源：https://stackoverflow.com/questions/32117471/how-do-i-measure-fragmentation-in-hotspots-metaspace