Why the site went down (or how to not send messages using camel)

Recently, we went down because of an issue that was hard to find.

The last release was the day before, so we didn’t directly suspect it to be release-related.

However, it was.
This is some new code that was in the release:

class UserActionNotifier(producerTemplate: org.apache.camel.ProducerTemplate, serialization: Serialization) {
 private[this] val userActionsEndpointUri: String = "activemq:topic:VirtualTopic.UserActions"

 def notifyOfUserAction(userId: Int, userActionType: String): Unit = {
 val ua = new UserAction(s"u$userId", 1, userActionType, new DateTime().getMillis)
 producerTemplate.sendBody(userActionsQueueEndpointUri, serialization.serialize(ua))
 }
}

Looks pretty innocuous, right?

Wrong.

What happens here is that we’re sending a message to a Virtual Topic in ActiveMQ.
When sending a message via a camel producerTemplate and passing in a String for the Endpoint, camel will do a lookup to get the actual endpoint to send to:

    protected Endpoint addEndpointToRegistry(String uri, Endpoint endpoint) {
        ObjectHelper.notEmpty(uri, "uri");
        ObjectHelper.notNull(endpoint, "endpoint");

        // if there is endpoint strategies, then use the endpoints they return
        // as this allows to intercept endpoints etc.
        for (EndpointStrategy strategy : endpointStrategies) {
            endpoint = strategy.registerEndpoint(uri, endpoint);
        }
        endpoints.put(getEndpointKey(uri, endpoint), endpoint);
        return endpoint;
    }

    protected EndpointKey getEndpointKey(String uri, Endpoint endpoint) {
        if (endpoint != null && !endpoint.isSingleton()) {
            int counter = endpointKeyCounter.incrementAndGet();
            return new EndpointKey(uri + ":" + counter);
        } else {
            return new EndpointKey(uri);
        }
    }

Source here and here (we are using Apache Camel 2.9.2).
As you can see, on line 15, it checks whether the Endpoint is a singleton. A Virtual Topic in ActiveMQ is not configured as a singleton, since its intent is to have multiple consumers.
However, when you’re sending messages, this goes wrong.

For each UserAction message that the service sends, a lookup is done to get the Endpoint and then a new EndpointKey is returned (with an incremented counter), which is added to the endpoints map (line 10).This leads to an ever-increasing map, which leads to the service blowing up and the site going down.

The following screenshots from Eclipse Memory Analyzer show the problem clearly:

Geert1

Geert2

In order to prevent this in the future, send messages using the actual Endpoint (instead of the endpoint name), like so:

class UserActionNotifier(producerTemplate: org.apache.camel.ProducerTemplate, context: org.apache.camel.CamelContext, serialization: Serialization) {

  private[this] val userActionsEndpoint: Endpoint = context.getEndpoint("activemq:topic:VirtualTopic.UserActions")

  def notifyOfUserAction(userId: Int, userActionType: String): Unit = {
    val ua = new UserAction(s"u$userId", 1, userActionType, new DateTime().getMillis)
    producerTemplate.sendBody(userActionsEndpoint, serialization.serialize(ua))
  }

}

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s