“Buffer pool is destroyed” issue found in Apache Flink flapMap Operator

问题

When I try to write to OUT collection in flatMap operator, I get illegal state exception(only under high load):Buffer pool is destroyed What wrong am I doing here ? When flink throws Buffer pool error ?

java.lang.RuntimeException:  Buffer pool is destroyed.

    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45)
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:696)
    at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)

..........
Caused by: java.lang.IllegalStateException:  Buffer pool is destroyed.
    at org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
    at org.apache.flink.runtime.io.network.buffer.BufferBuilder.append(BufferBuilder.java:79)
    at org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.copyToBufferBuilder(SpanningRecordSerializer.java:95)
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.copyFromSerializerToTargetChannel(RecordWriter.java:150)
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:128)
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:101)
    at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:81)
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:107)```

回答1:

What is a Buffer pool?

Buffer pool is a fixed size pool of memory segments instances for the network stack, which means that you put segments into it when you send data and fetch segments from it when you receive across the network.

Why is it destroyed?

The buffer pool is destroyd because the network environment shuts down.

Why does NetworkEnvironment shut down?

It may be a specific circumtances which I'm not very sure. I suggest you read more logs above this buffer pool exception.

来源：https://stackoverflow.com/questions/55468495/buffer-pool-is-destroyed-issue-found-in-apache-flink-flapmap-operator