g1gc

I'm running Spark 2 and am trying to shuffle around 5 terabytes of json. I'm running into very long garbage collection pauses during shuffling of a Dataset : val operations = spark.read.json(inPath).as[MyClass] operations.repartition(partitions, operations("id")).write.parquet("s3a://foo") Are there any obvious configuration tweaks to deal with this issue? My configuration is as follows: spark.driver.maxResultSize 6G spark.driver.memory 10G spark.executor.extraJavaOptions -XX:+UseG1GC -XX:MaxPermSize=1G -XX:+HeapDumpOnOutOfMemoryError spark.executor.memory 32G spark.hadoop.fs.s3a.buffer.dir

In Java 8 Update 45, adding these options to the java call: -XX:+PrintGCApplicationStoppedTime -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 shows me statistics like these: vmop [threads: total initially_running wait_to_block] [time: spin block sync cleanup vmop] page_trap_count 3679.229: no vm operation [ 72 1 2 ] [ 6016 0 6016 0 0 ] 1 2015-05-22T11:25:27.519+0200: Total time for which application threads were stopped: 6.0168551 seconds, Stopping threads took: 6.0164099 seconds The problem here is the long time for Stopping threads . In this example, it is 6 seconds which

I have hit counter running on Java. It works 24 hours and hits pass at about 100 times per second. During the day GC processing time slowly rises from 20-60 ms till 10000-60000 ms and then drops to 20-60 ms. Such pattern is repeated from time to time. From GC logs I found that mostly all time GC spend for processing Reference objects (Ref Proc). So what's the causes of such long GC time could be? Server: Amazon EC2 m1.small OS: Ubuntu 10.04.3 LTS Java: Oracle 1.7.0_07 Example of GC logs: 2012-09-13T16:51:20.091+0400: 167239.936: [GC pause (young), 62.58395400 secs] ... [Other: 62489.7 ms]

My server is using 1.8.0_92 on CentOS 6.7, GC param is '-Xms16g -Xmx16g -XX:+UseG1GC'. So the default InitiatingHeapOccupancyPercent is 45, G1HeapWastePercent is 5 and G1MixedGCLiveThresholdPercent is 85. My server's mixed GC starts from 7.2GB, but it clean less and less, finally old gen keeps larger than 7.2GB, so it's always try to do concurrent mark. Finally all heap are exhausted and full GC occurred. After full GC, old gen used is under 500MB. I'm curious why my mixed GC can't collect more, looks like live data is not so much... I have tried printing g1 related info, and found many

I'm currently having problems with very long garbage collection times. please see the followig. My current setup is that I'm using a -Xms1g and -Xmx3g. my application is using java 1.4.2. I don't have any garbage collection flags set. by the looks of it, 3gb is not enough and I really have a lot of objects to garbage collect. question: should I change my garbage collection algorithm? what should i use? is it better to use -XX:+UseParallelGC or -XX:+UseConcMarkSweepGC or should i use this combination -XX:+UseParNewGC -XX:+UseConcMarkSweepGC the ones occupying the memory are largely reports data