Webclient : java.lang.OutOfMemoryError: Direct buffer memory

问题

I am getting java.lang.OutOfMemoryError: Direct buffer memory error at the web client.

The batch job runs daily. It fails twice then passed in the third attempt.

at org.springframework.retry.support.RetryTemplate.rethrow(RetryTemplate.java:532) ~[spring-retry-1.2.5.RELEASE.jar:na]
 Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at reactor.netty.http.client.HttpClientDoOnError$OnErrorTcpClient.connect(HttpClientDoOnError.java:242) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]

JVM memory configuration which I found in the log. It is running in the cloud foundry.

JVM Memory Configuration: -Xmx342549K -Xss1M -XX:ReservedCodeCacheSize=240M -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=194026K

Code where it fails:

    @Retryable(maxAttemptsExpression = "#{${remote.retry.maxAttempts}}", backoff = @Backoff(delayExpression = "#{${remote.retry.delay}}"))
    public Optional<JobStatusResponseDTO> getStatus(String jobNumber, String accountNumber) {
        return broadridgeClient.getStatus(accountNumber, jobNumber);
    }

      @CircuitBreaker(maxAttemptsExpression = "#{${remote.circuitBreaker.maxAttempts}}",
            openTimeoutExpression = "#{${remote.circuitBreaker.openTimeout}}", resetTimeoutExpression = "#{${remote.circuitBreaker.resetTimeout}}")
    public Optional<JobStatusResponseDTO> getStatus(String account, String jobNumber) {

        JobStatusRequestDTO request = new JobStatusRequestDTO();
        request.setAccount(account);
        request.setJobNumber(jobNumber);

        JobStatusResponseDTO jobStatus;
        jobStatus = client.post()
                .uri(PATH)
                .body(BodyInserters.fromValue(request))
                .exchange()
                .elapsed()
                .flatMap(response -> {
                    
                    if (response.getT2().statusCode() == HttpStatus.NO_CONTENT) {
                        return Mono.empty();
                    } else if (isClientOrServerError(response.getT2())) {
                        return Mono.error(new RemoteClientException(String.format("Job status is not received: %s", response.getT2().statusCode())));
                    }
                    return response.getT2().bodyToMono(JobStatusResponseDTO.class);
                })
                .block();
        return Optional.ofNullable(jobStatus);
    }

Edit 1:

I also found that cloud foundary is setting a low mwmory for heap I am using CF task to run it.

cf run-task ipbol-proxy-batch "JAVA_OPTS=\"-agentpath:\$PWD/.java-buildpack/oracle_jre/bin/jvmkill-1.16.0_RELEASE=printHeapHistogram=1 -Djava.io.tmpdir=\$TMPDIR -Djava.security.egd=file:///dev/urandom -XX:ActiveProcessorCount=\$(nproc) -Dspring.batch.job.names=${job_name} -Dbatch.run.historic=${run_historic_files} -Dorg.cloudfoundry.security.keymanager.enabled=false -Dorg.cloudfoundry.security.trustmanager.enabled=true -Djava.ext.dirs= -Djava.security.properties=\$PWD/.java-buildpack/java_security/java.security \$JAVA_OPTS\" && CALCULATED_MEMORY=\$(\$PWD/.java-buildpack/oracle_jre/bin/java-buildpack-memory-calculator-3.13.0_RELEASE -totMemory=\$MEMORY_LIMIT -loadedClasses=31842 -poolType=metaspace -stackThreads=250 -vmOptions=\"\$JAVA_OPTS\") && echo JVM Memory Configuration: \$CALCULATED_MEMORY && JAVA_OPTS=\"\$JAVA_OPTS \$CALCULATED_MEMORY\" && KEYSTORE_PARAMS=\$(\$PWD/.java-buildpack/oracle_jre/bin/java -jar \$PWD/.java-buildpack/oracle_jre/keystore-manager-0.1.1.jar) && JAVA_OPTS=\"\$JAVA_OPTS \$KEYSTORE_PARAMS\" && MALLOC_ARENA_MAX=2 VCAP_SERVICES=\$(for i in {1..10}; do \${PWD}/.java-buildpack/mv_decryptor/jpmc-mvdecryptor && exit 0; sleep 3; done; kill \$\$) SERVER_PORT=\$PORT eval exec \$PWD/.java-buildpack/oracle_jre/bin/java \$JAVA_OPTS -cp \$PWD/.::\$PWD/.java-buildpack/container_security_provider/container_security_provider-1.16.0_RELEASE.jar org.springframework.boot.loader.JarLauncher" ipbol-proxy-batch

Edit 2:

I have updated the code to consume the response in every case. Still, I am getting the same "java.lang.OutOfMemoryError: Direct buffer memory"

public Optional<JobStatusResponseDTO> getStatus(String account, String jobNumber) {

        JobStatusRequestDTO request = new JobStatusRequestDTO();
        request.setAccount(account);
        request.setJobNumber(jobNumber);
        Optional<JobStatusResponseDTO> responseDTO = Optional.empty();

        final Object block = client.post()
                .uri(PATH)
                .body(BodyInserters.fromValue(request))
                .exchange()
                .elapsed()
                .flatMap(response -> {
                   
                    if (response.getT2().statusCode() == HttpStatus.NO_CONTENT) {
                        return response.getT2().bodyToMono(Void.class).thenEmpty(Mono.empty());

                    } else if (isClientOrServerError(response.getT2())) {
                        return response.getT2().bodyToMono(Void.class).thenEmpty(Mono.error(new RemoteClientException(String.format("Job status is not received: %s", response.getT2().statusCode()))));
                    }
                    return response.getT2().bodyToMono(JobStatusResponseDTO.class);
                })
                .block();
        
        if(block!= null && block instanceof JobStatusResponseDTO)
        {
            responseDTO = Optional.of((JobStatusResponseDTO)block);
        }
        return responseDTO;
    }

I don't have spring webflux 5.3 in organization repo. So can't use that.

This is the default memory configuration

-Xmx1387828K -Xss1M -XX:ReservedCodeCacheSize=240M -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=197323K

回答1:

Since Spring 5.3 the exchange() method has been deprecated on the web client due to the fact it opens up the possibility of memory and connection leaks. By using exchange() you take on the responsibility to consume the response content for every single scenario.

From the code example above, you are not consuming the body in success and error cases.

You can also set the following property to get some additional information from netty on where the leaks are coming from.