Spring AMQP v1.4.2 - Rabbit reconnection issue on network failure

前端 未结 3 1051
自闭症患者
自闭症患者 2021-01-08 01:10

I am testing the following scenario in Spring AMQP v1.4.2 and it fails to reconnect after a network disruption:

  1. Start the spring application which consumes mess
相关标签:
3条回答
  • 2021-01-08 01:45

    I just ran your test as described (rabbit on linux using iptables to drop packets).

    There is no log when the connection is reestablished (perhaps we should).

    I suggest you turn on debug logging to see the reconnection.

    EDIT:

    From the rabbitmq documentation:

    exclusive Exclusive queues may only be accessed by the current connection, and are deleted when that connection closes. Passive declaration of an exclusive queue by other connections are not allowed.

    From your exception:

    reply-code=405, reply-text=RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'e4288669-2422-40e6-a2ee-b99542509273' in vhost '/', class-id=50, method-

    So the problem is the broker still thinks the other connection exists.

    1. Don't use an exclusive queue (you will lose messages with such a queue anyway). Or,
    2. Set a low requestedHeartbeat so the broker will detect the lost connection faster.
    0 讨论(0)
  • 2021-01-08 01:52

    Set setRequestedHeartBeat to the ConnectionFactory and setMissingQueuesFatal(false) to SimpleMessageListenerContainer in order to retry to connect indefinitely. By default SimpleMessageListenerContainer setMissingQueuesFatal is set to true and only 3 retries will be done.

      @Bean
      public ConnectionFactory connectionFactory() {
        final CachingConnectionFactory connectionFactory = new CachingConnectionFactory(getHost(), getPort());
        connectionFactory.setUsername(getUsername());
        connectionFactory.setPassword(getPassword());
        connectionFactory.setVirtualHost(getVirtualHost());
        connectionFactory.setRequestedHeartBeat(30);
        return connectionFactory;
      }
    
      @Bean
      public SimpleMessageListenerContainer listenerContainerCopernicusErrorQueue() {
        final SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
        container.setConnectionFactory(connectionFactory());
        container.setQueueNames(myQueue().getName());
        container.setMessageListener(messageListenerAdapterQueue());
        container.setDefaultRequeueRejected(false);
        container.setMissingQueuesFatal(false);
        return container;
      }
    
    0 讨论(0)
  • 2021-01-08 01:55

    We are also facing this issue in our production environment as well, might be because of Rabbit nodes running as VMs on different ESX racks etc. The workaround that we found was to have our client app continuously keep trying to reconnect if it gets disconnected from cluster. Below are the settings we applied and it worked:

    <util:properties id="spring.amqp.global.properties">
      <prop key="smlc.missing.queues.fatal">false</prop>
    </util:properties>
    

    This attribute changes the global behavior of Spring AMQP when declaring queues fails for fatal errors (broker not available etc). By default container tries 3 times only (see log message showing "retries left=0").

    Ref: http://docs.spring.io/spring-amqp/reference/htmlsingle/#containerAttributes

    In addition, we added recovery-interval so that container recovers from non-fatal errors. However, the same config is also used when global behavior is to retry for fatal errors too (like missing queues).

    <rabbit:listener-container recovery-interval="15000" connection-factory="consumerConnectionFactory">
    ....
    </rabbit:listener-container>
    
    0 讨论(0)
提交回复
热议问题