Huge heap dump (11GB) - Jhat failed & Eclipse MAT needed help

Question

We got a memory error in our EA and we used -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/some/dir to dump the heap at the time of OOM.

We had a heap memory of 12GB and perm gen of 256MB.

The heap dump is generated in the Linux box where we have our application running and its size is 11.5GB. We don't have privileges to download it to our local.

When we tried to analyze that 11GB heap dump using JHAT it thrown an OOM.

We tried the following commands from our Linux CLI.

jhat java_pid1491.hprof
jhat -J-Xmx16g -XX:-UseBiasedLocking java_pid1491.hprof
jhat -J-d64 -J-Xmx16g -J-XX:-UseBiasedLocking java_pid1491.hprof#1

For all the options its throwing an OOM exception after reading the dump for several minutes(>30 mins).

We googled it and found MAT as a powerful heap dump analyzer, but not a way to use it in LINUX.

Any suggestions would be of greater help. Thanks.

Modified:

Got MAT installed in Linux x86_64 machine but got following error when executed ./MemoryAnalyzer

(.:17319): GLib-GObject-WARNING **: invalid (NULL) pointer instance
(.:17319): GLib-GObject-CRITICAL **: g_signal_connect_data: assertion `G_TYPE_CHECK_INSTANCE (instance)' failed
(.:17319): Gtk-CRITICAL **: gtk_settings_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(.:17319): GLib-GObject-CRITICAL **: g_object_get: assertion `G_IS_OBJECT (object)' failed
(.:17319): Gtk-WARNING **: Screen for GtkWindow not set; you must always set
a screen for a GtkWindow before using the window
(.:17319): Gdk-CRITICAL **: gdk_pango_context_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(.:17319): Pango-CRITICAL **: pango_context_set_font_description: assertion `context != NULL' failed
(.:17319): Pango-CRITICAL **: pango_context_set_base_dir: assertion `context != NULL' failed
(.:17319): Pango-CRITICAL **: pango_context_set_language: assertion `context != NULL' failed
(.:17319): Pango-CRITICAL **: pango_layout_new: assertion `context != NULL' failed
(.:17319): Pango-CRITICAL **: pango_layout_set_text: assertion `layout != NULL' failed
(.:17319): Pango-CRITICAL **: pango_layout_set_attributes: assertion `layout != NULL' failed
(.:17319): Pango-CRITICAL **: pango_layout_set_alignment: assertion `layout != NULL' failed
(.:17319): Pango-CRITICAL **: pango_layout_set_ellipsize: assertion `PANGO_IS_LAYOUT (layout)' failed
(.:17319): Pango-CRITICAL **: pango_layout_set_single_paragraph_mode: assertion `PANGO_IS_LAYOUT (layout)' failed
(.:17319): Pango-CRITICAL **: pango_layout_set_width: assertion `layout != NULL' failed
(.:17319): Pango-CRITICAL **: pango_layout_get_extents: assertion `layout != NULL' failed
(.:17319): Gtk-CRITICAL **: gtk_icon_theme_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(.:17319): Gtk-CRITICAL **: gtk_settings_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(.:17319): Gtk-CRITICAL **: gtk_icon_size_lookup_for_settings: assertion `GTK_IS_SETTINGS (settings)' failed
(.:17319): Gtk-WARNING **: Invalid icon size 6
(.:17319): Gtk-CRITICAL **: gtk_icon_theme_load_icon: assertion `GTK_IS_ICON_THEME (icon_theme)' failed
Segmentation fault

I tried ./ParseHeapDump.sh ../java_pid1491.hprof this and got following error,

Caused by: java.lang.NoClassDefFoundError: com/ibm/dtfj/image/CorruptDataException
        at org.eclipse.mat.dtfj.InitDTFJ.stop(InitDTFJ.java:70)

Answer 1

I was using MAT on Linux yesterday.... check out the proper version here

Answer 2

forget Jhat, you would need more than 100Gybte of RAM to parse this dump, it would take forever and it would be very hard to find something. I just lately installed MAT on a SUSE 11 machine. worked flawlessly . Can you please make sure that MAT is using the right JVM? Not sure whether we testet it with anything else then the SUN/SAP JVMs.

Regards, Markus (kohlerm)

Answer 3

We have had success in the past using Netbeans to analyze OOM errors. It seemed to cope where Eclipse and JHAT couldn't.