Detect and recover from credential deletion in Spring AMQP

问题

We have a Spring Cloud Config setup, using Vault database backends (MySQL and RabbitMQ), which gives us the ability to inject generated credentials into properties such as:

• spring.rabbitmq.username
• spring.rabbitmq.password

When our app starts up we have a fresh set of Rabbit credentials, and we have the ability to request a new set on demand.

Since our Rabbit credentials are managed externally by Vault, they could be expired / deleted at any time during the app's life (this is also a resilience test scenario).

My question is how we can (effectively, reliably):

• Detect the expiry of the generated credentials
• Update our existing Spring AMQP CachingConnectionFactory with the new credentials.

We're working on the basis that this needs to be handled entirely client-side as a matter of resilience, even if the server was willing or able to send expiry notifications.

What we're struggling with is how to detect credential expiry so that we can then reconfigure the CachingConnectionFactory.

Possibilities include:

1. What we have now: a ChannelListener that builds a list of all newly-created Channels, and tries creating/deleting an anonymous Queue on each every x seconds, listening out for any ShutdownSignalExceptions via ShutdownListener that might have a 403 status code. This seems to work but is a bit complex, and we've seen concurrency issues doing anything non-trivial in the shutdown handler.
2. Hook into CachingConnectionFactory somehow. We tried working with a clone of the class but besides the complexity of that we just ended up with RESOURCE_LOCKED errors creating queues.
3. Something simpler and more lightweight, e.g. simply poll the broker every x seconds to validate the current credentials still exist.

Part of the problem is that ACCESS_REFUSED - what you get when CachingConnectionFactory tries to work with deleted credentials - is generally treated as a fatal misconfiguration error rather than part of any real workflow, or that can possibly be recovered from.

Is there a graceful solution here?

---

Using: Spring Boot 1.5.10-RELEASE, Spring Cloud Dalston SR4

---

Update:

On the RabbitTemplate side, no exception is thrown - with or without RetryTemplate - even when CachingConnectionFactory correctly detects ACCESS_REFUSED to the exchange that I'm sending to.

Configuration is:

spring
  rabbitmq:
    host: rabbitmq.service.consul
    port: 5672
    virtualHost: /
    template:
      retry:
        enabled: true

Code is:

@Autowired private RabbitTemplate rt;  // From RabbitAutoConfiguration

@Bean
public DirectExchange emailExchange() {
    return new DirectExchange("email");
}

public void sendEmail() {
    this.rt.send("email", "email.send", "test payload");
}

Application starts up, declaring the email exchange. The RabbitMQ UI shows my (generated) user and connection to the exchange, which is fine at startup. I then simulate credential expiry by manually deleting that user using the UI before running a local test to call sendEmail() email above.

No exceptions are thrown or errors logged as a result of the RabbitTemplate call, but the following (expected) error is logged:

[AMQP Connection 127.0.0.1:5672] ERROR o.s.a.r.c.CachingConnectionFactory - Channel shutdown: channel error; protocol method: #method(reply-code=403, reply-text=ACCESS_REFUSED - access to exchange 'email' in vhost '/' refused for user 'cert-configserver-75c3ae60-da76-3058-e7df-a7b90ef72171', class-id=60, method-id=40)

Short of checking credentials before all RabbitTemplate.send() calls, I'd like to know if there's any way to catch the ACCESS_REFUSED error during send so that I can refresh credentials as I do for listeners, and give RetryTemplate a chance to retry.

回答1:

For such a scenario a listener container emits a ListenerContainerConsumerFailedEvent. You can listen to this one, check its reason and exception and decide to stop() the container and do something else you need. Then start() it again to consume the broker with fresh credentials.

On the RabbitTemplate side there is just need to try...catch the call and analyze an exception for the same reason.

That's not what I tried so far, but this is my best feeling how to deal with the ACCESS_REFUSED state. You really can't do anything on the matter from the CachingConnectionFactory perspective.

UPDATE

My application is like this:

spring.rabbitmq.username=test
spring.rabbitmq.password=test
spring.rabbitmq.template.retry.enabled=true
spring.rabbitmq.template.retry.initial-interval=1ms
logging.level.org.springframework.retry=DEBUG

---

@SpringBootApplication
public class So49155945Application {

    public static void main(String[] args) {
        ConfigurableApplicationContext applicationContext = SpringApplication.run(So49155945Application.class, args);
        RabbitTemplate rabbitTemplate = applicationContext.getBean(RabbitTemplate.class);

        try {
            rabbitTemplate.convertAndSend("foo", "foo");
        }
        catch (AmqpException e) {
            System.err.println("Error during sending: " + e.getCause().getCause().getMessage());
        }
    }

}

And this is what I have in the console when I run this app for that non-existing user:

Error during sending: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.

UPDATE 2

What I found also as we can make these props:

spring.rabbitmq.publisher-confirms=true
spring.rabbitmq.template.mandatory=true

And then add a rabbitTemplate.setConfirmCallback() and our rejected message for the async send will be rejected. However it still is a async callback, similar to the mentioned ChannelListener. There is really nothing to do from the Spring AMQP perspective. Everything is an async nature of the AMQP protocol and might really need some "fail fast" hook from the Rabbit Client library.

Please, ask such a question on the rabbitmq-users Google group. That's the place where RabbitMQ engineers hang outs.

UPDATE 3

As the solution for such events on the Broker the Event Exchange Plugin can be used. The particular user.deleted or user.password.changed events are emitted by the Broker.

回答2:

After much experimentation and debugging, I took Artem Bilan's suggestion and adopted the RabbitMQ Event Exchange Plugin.

So now, instead of trying to track ShutdownSignalException and ListenerContainerConsumerFailedEvent events across Spring and Rabbit code, between SimpleMessageListenerContainer on one hand, and RabbitTemplate on the other, I simply subscribe to an exchange and let my new @RabbitListener notify me of credential issues. This without any other moving parts or bean declarations, without any synchronization issues or blocked threads, and generally going with the flow of autoconfiguration rather than fighting it.

All I now need is:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.amqp.rabbit.annotation.*;
import org.springframework.amqp.rabbit.connection.CachingConnectionFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.amqp.RabbitProperties;
import org.springframework.cloud.endpoint.RefreshEndpoint;
import org.springframework.messaging.MessageHeaders;
import org.springframework.stereotype.Component;

import static org.springframework.amqp.core.ExchangeTypes.TOPIC;

@Component
public class ReuathenticationListener {

    private static Logger log = LoggerFactory.getLogger(ReuathenticationListener.class);

    @Autowired private RabbitProperties rabbitProperties;
    @Autowired private RefreshEndpoint refreshEndpoint;
    @Autowired private CachingConnectionFactory connectionFactory;

    @RabbitListener(
        id = "credential_expiry_listener",
        bindings = @QueueBinding(value = @Queue(value="credentials.expiry", autoDelete="true", durable="false"),
            exchange = @Exchange(value="amq.rabbitmq.event", type=TOPIC, internal="true", durable="true"),
            key = "user.#")
    )
    public void expiryHandler(final MessageHeaders headers) {
        final String key = (String) headers.get("amqp_receivedRoutingKey");
        // See: https://www.rabbitmq.com/event-exchange.html
        if (!key.equals("user.deleted") &&
            !key.equals("user.authentication.failure")) {
            return;
        }

        final String failedName = (String) headers.get("name");
        final String prevUsername = rabbitProperties.getUsername();

        if (!failedName.equals(prevUsername)) {
            log.debug("Ignore expiry of unrelated user: " + failedName);
            return;
        }

        log.info("Refreshing Rabbit credentials...");
        refreshEndpoint.refresh();
        log.info("Refreshed username: '" + prevUsername + "' => '" + rabbitProperties.getUsername() + "'");

        connectionFactory.setUsername(rabbitProperties.getUsername());
        connectionFactory.setPassword(rabbitProperties.getPassword());
        connectionFactory.resetConnection();

        log.info("CachingConnectionFactory reset, reconnection should now begin.");
    }
}

来源：https://stackoverflow.com/questions/49155600/detect-and-recover-from-credential-deletion-in-spring-amqp