1. 技术文章
  2. Understanding Jetty's “Closed while Pending/Unready” warning

Understanding Jetty's “Closed while Pending/Unready” warning

由 筅森魡賤 提交于 2019-12-06 04:45:38

It is possible to reproduce the Closed while Pending/Unready warning with some manual debugging and a plain curl client. I have tested it with Jetty 9.4.8.v20171121 and Jetty 9.4.11.v20180605. You need two breakpoints, so it's not easy to reliably reproduce in a test environment.

  1. The first breakpoint is in the HttpOutput.write() method right after it changes its state from READY to PENDING:

    case READY:
    if (!_state.compareAndSet(OutputState.READY, OutputState.PENDING))
        continue;
    // put a breakpoint here

  2. The second one in Response.closeOutput():

    case STREAM:
    // put a breakpiont here
    if (!_out.isClosed())
        getOutputStream().close();
    break;

Steps to reproduce:

  1. [JettyThread1] Calls QueueWriteListener.onWritePossible() which writes a few bytes to the output then returns (as its input buffer is empty).
  2. Wait for the onTimeout event.
  3. [JettyThread2] HttpChannelState.onTimeout() calls QueueWriteListener.onTimeout() which calls asyncContext.complete().
  4. [JettyThread2] HttpChannelState.onTimeout() schedules a dispatch after the async timeout.
  5. [JettyThread2] Pauses at the second breakpoint
  6. [DFT] DataFeederThread calls writeImpl()
  7. [DFT] DataFeederThread calls HttpOutput.write() (it's the output stream)
  8. [DFT] HttpOutput.write() changes its state from READY to PENDING
  9. [DFT] DataFeederThread pauses here, due to the breakpoint above
  10. [JettyThread2] The scheduled dispatch closes the output stream and produces the Closed while Pending/Unready warning.

So, actually it's Jetty who closes the output stream on this (Jetty 9.4.8.v20171121) stack:

Thread [jetty-19] (Suspended)   
    Response.closeOutput() line: 1043   
    HttpChannelOverHttp(HttpChannel).handle() line: 488 
    HttpChannelOverHttp(HttpChannel).run() line: 293    
    QueuedThreadPool.runJob(Runnable) line: 708 
    QueuedThreadPool$2.run() line: 626  
    Thread.run() line: 748

Making onTimeout() synchronized (as well as writeImpl() is also synchronized) in the listener does not help since the scheduled closing still be able to overlap with writeImpl (from DataFeederThread). Consider this case:

  1. [JettyThread1] Calls QueueWriteListener.onWritePossible() which writes a few bytes to the output then returns (as its input buffer is empty).
  2. Wait for the onTimeout event.
  3. [JettyThread2] HttpChannelState.onTimeout() calls QueueWriteListener.onTimeout() which calls asyncContext.complete().
  4. [DFT] DataFeederThread calls writeImpl() (it's blocked since onTimeout has not finished yet)
  5. [JettyThread2] QueueWriteListener.onTimeout() finishes
  6. [DFT] writeImpl() can run
  7. [JettyThread2] HttpChannelState.onTimeout() schedules a dispatch after the async timeout.
  8. [JettyThread2] Pauses at the second breakpoint
  9. [DFT] DataFeederThread calls HttpOutput.write() (it's the output stream)
  10. [DFT] HttpOutput.write() changes its state from READY to PENDING
  11. [DFT] DataFeederThread pauses here, due to the breakpoint above
  12. [JettyThread2] The scheduled dispatch closes the output stream and produces the Closed while Pending/Unready warning.

Unfortunately, after asnyContext.complete() it is not enough to check output.isReady(). It returns true since Jetty reopens the HttpOutput (see the stack below), so you need a separate flag for that in the listener.

Thread [jetty-13] (Suspended (access of field _state in HttpOutput))    
    HttpOutput.reopen() line: 195   
    HttpOutput.recycle() line: 923  
    Response.recycle() line: 138    
    HttpChannelOverHttp(HttpChannel).recycle() line: 269    
    HttpChannelOverHttp.recycle() line: 83  
    HttpConnection.onCompleted() line: 424  
    HttpChannelOverHttp(HttpChannel).onCompleted() line: 695    
    HttpChannelOverHttp(HttpChannel).handle() line: 493 
    HttpChannelOverHttp(HttpChannel).run() line: 293    
    QueuedThreadPool.runJob(Runnable) line: 708 
    QueuedThreadPool$2.run() line: 626  
    Thread.run() line: 748

Furthermore, isReady() also returns true when the output is still closed (before recycle/reopen). Related question: isReady() returns true in closed state - why?

The final implementation is something similar:

@Override
protected void doPost(final HttpServletRequest req, final HttpServletResponse resp)
        throws ServletException, IOException {
    resp.setContentType(MediaType.OCTET_STREAM.type());
    resp.setStatus(HttpServletResponse.SC_OK);
    resp.setBufferSize(4096);
    resp.flushBuffer();
    final AsyncContext async = req.startAsync();
    async.setTimeout(5_000); // millis
    final ServletOutputStream output = resp.getOutputStream();
    final QueueWriteListener writeListener = new QueueWriteListener(async, output);
    async.addListener(writeListener);
    output.setWriteListener(writeListener);
}

private static class QueueWriteListener implements AsyncListener, WriteListener {

    private static final Logger logger = LoggerFactory.getLogger(QueueWriteListener.class);

    private final AsyncContext asyncContext;
    private final ServletOutputStream output;

    @GuardedBy("this")
    private boolean completed = false;

    public QueueWriteListener(final AsyncContext asyncContext, final ServletOutputStream output) {
        this.asyncContext = checkNotNull(asyncContext, "asyncContext cannot be null");
        this.output = checkNotNull(output, "output cannot be null");
    }

    @Override
    public void onWritePossible() throws IOException {
        writeImpl();
    }

    private synchronized void writeImpl() throws IOException {
        if (completed) {
            return;
        }
        while (output.isReady()) {
            final byte[] message = getNextMessage();
            if (message == null) {
                output.flush();
                return;
            }
            output.write(message);
        }
    }

    private synchronized void completeImpl() {
        // also stops DataFeederThread to call bufferArrived
        completed = true;
        asyncContext.complete();
    }

    @Override
    public void onError(final Throwable t) {
        logger.error("Writer.onError", t);
        completeImpl();
    }

    public void dataArrived() {
        try {
            writeImpl();
        } catch (RuntimeException | IOException e) {
            ...
        }
    }

    public void noMoreData() {
        completeImpl();
    }

    @Override
    public synchronized void onComplete(final AsyncEvent event) throws IOException {
        completed = true; // might not needed but does not hurt
    }

    @Override
    public synchronized void onTimeout(final AsyncEvent event) throws IOException {
        completeImpl();
    }

    @Override
    public void onError(final AsyncEvent event) throws IOException {
        logger.error("onError", event.getThrowable());
    }

    ...
}

update 2018-08-01: Actually it did not fix the warning completely, see: “Closed while Pending/Unready” warnings from Jetty

You are using Asynchronous I/O from the Servlet Spec and manually closed the Response stream.

Key Fact: A close call implies a write operation.

In the Jetty case, the IOException is telling you that your manual call to the Response stream close has caused an undesired side-effect.

So in Asynchronous I/O mode a call to ServletOutputStream.isReady() should have been made prior to using ServletOutputStream.close() verifying that isReady() is true.

Note that it can be desirable to allow a ServletOutputStream.close() at any time, particularly if AsyncContext.complete() has been called on this dispatch.

However, if the prior write has not yet completed and/or ServletOutputStream.isReady() has not been called, this specific case of a naked ServletOutputStream.close() is allowed, but the behavior is to abort the response, discarding any pending/unwritten writes on the ServletOutputStream.

This is why you get the IOException("Closed while Pending/Unready")

Now you should ask yourself why are you closing the Response stream? This is not required per the Servlet spec, and is undesired if you ever wanted to use the RequestDispatcher to composite multiple Servlets responses together.

标签
servlets
jetty
embedded-jetty
servlet-3.1
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!