Java thread blocking

Question

i have a problem with my java environement. I\'m running Solr 1.3 (search engine) since more then a year now and suddenly i got alot of trouble with it. All my thread pool

Answer 1

All your Threads are logging things. They all need to write on the disk from time to time. Everytime one of your 240 threads hits a logging line there will be disk access issues.

It baffles me that the Thread having the lock is in the RUNNABLE state.

I think it might be waiting for some external ressource to be released (like disk access for example)

Are you running low on disk space? Have you recently changed something in your storage system?

Answer 2

If you are running under Windows and the java application starts a console, be careful not to click in the DOS box. Window's crappy mark and copy "feature" blocks output to the ConsoleHandler. So any logger trying to write to the screen will block. Writing to the Console is done in a native call and so the java thread will appear to be in a RUNNING state when in fact it is blocked, it's just there's no way to feed that blocked status back to the application (because you are in native space).

If the application is blocked (you have clicked in the DOS box) press escape to continue.

Answer 3

According to your log, the issue concerns the use of java.util.logging.ConsoleHandler.

First try to disable the console handler by removing it from the 'handlers' and '.handlers' list in '${TOMCAT_HOME}/conf/logging.properties'. See whether the problem still occurs.

If that helps, than it is definitely an issue with the output of the ConsoleHandler. Try checking whether there is are issues concerning the 'catalina.out' file. This is the file to where tomcat redirects its console output.

Answer 4

It seems that the thread that own "0x00007fe37e72b340" is blocked at the IO level. Maybe a disk (raid?) issue?

can you do a thread dump 5 minutes later the see if the same thread is still blocked?

Answer 5

Flushing after each log record is going to expensive if you have very verbose logs.

A quality fix would be to clean up the logging, probably based around auditing.

As a quick fix, override StreamHandler.flush or OutputStream.flush to not do so immediately. Only flush once every so often. Note, however that you could potentially lose logging data immediately before a crash if you do this.

Answer 6

I never used java.util.logging, so I don't know whether my suggestion is useful, but netherless:
try to use different instance of java.util.logging.Logger, so not all 240 threads will be blocked on the same monitor
(it will help if different instances of Logger use different instances of java.util.logging.ConsoleHandler) .