I am testing the following scenario in Spring AMQP v1.4.2 and it fails to reconnect after a network disruption:
I just ran your test as described (rabbit on linux using iptables
to drop packets).
There is no log when the connection is reestablished (perhaps we should).
I suggest you turn on debug logging to see the reconnection.
EDIT:
From the rabbitmq documentation:
exclusive Exclusive queues may only be accessed by the current connection, and are deleted when that connection closes. Passive declaration of an exclusive queue by other connections are not allowed.
From your exception:
reply-code=405, reply-text=RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'e4288669-2422-40e6-a2ee-b99542509273' in vhost '/', class-id=50, method-
So the problem is the broker still thinks the other connection exists.
requestedHeartbeat
so the broker will detect the lost connection faster.Set setRequestedHeartBeat
to the ConnectionFactory
and setMissingQueuesFatal(false)
to SimpleMessageListenerContainer
in order to retry to connect indefinitely. By default SimpleMessageListenerContainer setMissingQueuesFatal is set to true and only 3 retries will be done.
@Bean
public ConnectionFactory connectionFactory() {
final CachingConnectionFactory connectionFactory = new CachingConnectionFactory(getHost(), getPort());
connectionFactory.setUsername(getUsername());
connectionFactory.setPassword(getPassword());
connectionFactory.setVirtualHost(getVirtualHost());
connectionFactory.setRequestedHeartBeat(30);
return connectionFactory;
}
@Bean
public SimpleMessageListenerContainer listenerContainerCopernicusErrorQueue() {
final SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory());
container.setQueueNames(myQueue().getName());
container.setMessageListener(messageListenerAdapterQueue());
container.setDefaultRequeueRejected(false);
container.setMissingQueuesFatal(false);
return container;
}
We are also facing this issue in our production environment as well, might be because of Rabbit nodes running as VMs on different ESX racks etc. The workaround that we found was to have our client app continuously keep trying to reconnect if it gets disconnected from cluster. Below are the settings we applied and it worked:
<util:properties id="spring.amqp.global.properties">
<prop key="smlc.missing.queues.fatal">false</prop>
</util:properties>
This attribute changes the global behavior of Spring AMQP when declaring queues fails for fatal errors (broker not available etc). By default container tries 3 times only (see log message showing "retries left=0").
Ref: http://docs.spring.io/spring-amqp/reference/htmlsingle/#containerAttributes
In addition, we added recovery-interval so that container recovers from non-fatal errors. However, the same config is also used when global behavior is to retry for fatal errors too (like missing queues).
<rabbit:listener-container recovery-interval="15000" connection-factory="consumerConnectionFactory">
....
</rabbit:listener-container>