Ehcache replicated caching with JMS, AWS, SQS, SNS & Nevado

Read part 2 of this research here

Recently I’ve been researching ways to GSLB a very large app that relies on Ehcache for numerous things internally such as; cached page output, cached lists etc etc. Anyone who has experience getting large scale applications to function active-N in a GSLB’d setup knows the challenges such topologies present. The typical challenge you will face is how to bridge events that occur in locally (dc) clustered applications, for example in: DC-A (data center), with another logical instance footprint of the same application living in DC-B. This extends all the way from the from the data-source, all the way up the stack.

So for example, lets say user A is accessing the application and hitting instances of it residing in DC-A. This user updates some inventory data that is cached locally in the cluster in DC-A; subsequently this cached inventory also resides in the cluster running in DC-B (also being access by different users in some other region). When user A updates this inventory data, the local application instance, writes it to the data-source, and then does some sort of cache operation, such as a cache key remove, or put (replace). Forgetting the entirely separate issue of how the data-source write is itself is visible across both DC’s, point being is that the cache update in DC-A is visible only to participating instances in DC-A….. DC-B’s cache knows nothing of this; only its data-source is aware of this new information…. so we need a way to get DC-B aware this cache event. There are a few ways this can happen; for example we could just configure the caches to rely solely or LRU/TTL driven expiry, or actually respond to events in a near-real-time fashion.

Now before we go on I’ll state up-front that despite what I am about to describe would work (to an extent), ultimately I likely will NOT go with this setup due to the inefficiencies involved, particularly the amount of data being moved across WANs if you just use the Ehcache JMS replicated caching feature alone. (i.e. cached data is actually moved around, rather than just operation events with the JMS replicated Ehcache feature)

Continuing with that train of thought, after the latter caveat…. so one thing I started looking at was the Ehcache JMS Replicated Caching feature. Basically this feature boils down to permitting you to configure any cache to utilize JMS (Java message service) for publishing cache events. So when a PUT/REMOVE happens, Ehcache wires up a cache listener that responds and subsequently relays these events (including the cached data on puts) to a JMS topic. Then any other Ehcache node configured w/ this same setup can subscribe to those topics and receive those events. Couple this with a globally accessible messaging system, you now can have a backbone for distributing these events across multiple disparate data-centers…… but who in the hell wants to setup their own globally accessible, fault-tolerant messaging system implementation…. not me.

Enter AWS’s SNS (Simple Notification Service,  topics) & SQS (Simple queuing service) services. I decided I’d try to get Ehcache’s JMS Replicated Caching feature to utilize AWS as the JMS provider….. now enter Nevado JMS from the Skyscreamer Team. (github link). Nevado is a cool little JMS implementation that front’s SNS/SQS, and it works pretty good!

Note the code is at the end of this post ….. and yes the code is very basic and NOT production ready/tested; it was just for a prototype/research and is a bit hacked together. Also note this code is reliant upon this PATCH to Nevado, which is pending discussion/approval

  1. The first step was creating an Ehcache CacheManagerPeerProviderFactory (NevadoJMSCacheManagerPeerProviderFactory), which returns a JMSCacheManagerPeerProvider to Ehcache that is configured to use Nevado on the backend
  2. The NevadoJMSCacheManagerPeerProviderFactory boots a little spring context that sets up the NevadoTopic etc
  3. Created a little test program (below) EhcacheNevadoJMSTest. I just ran several instances of this concurrently w/ breakpoints to validate that events in one JVM/ehcache instance were indeed being broadcast over JMS -> AWS -> back to other Ehcache instances on other JVM instances.
  4. The first thing I noticed was that while the events were indeed being sent over JMS to AWS and received by other Ehcache peers, the actual cached data (Element) embedded within the JMSEventMessage were NOT being included, resulting in NullPointerException’s by the Ehcache peers who received the event.
  5. The latter was due to an Object serialization issue, and transient soft references as described in this Nevado Github issue #81
  6. Once I created a patch for Nevado to use the ObjectOutputStream things worked perfectly.

CAVEATS

  • Again this code was for research/prototyping
  • The viability of having the actual cached element being moved around to AWS, across WANs and back to other data-centers is likely not too optimal. It would work, but under high-volume you could spend a lot of $$ and bandwidth.
  • SQS/SNS has message size limitations…. which if your cached data is beyond that would get truncated and lost effectively making the solution useless.
  • Ideally, all one really cares about is “what happened”, meaning Ehcache KEY-A was PUT or REMOVED etc. Then let the receiving DC decide what to do (i.e. remove the cached KEY locally and let next user driven request re-populated from the primary source, the real data-source). This results in much smaller message sizes. The latter is what I’m now looking at, using the Ehcache listener framework w/ some custom calls to SNS/SQS would suffice for this kind of implementation.

 

 

CODE SNIPPETS

Github patch for Nevado @ https://github.com/skyscreamer/nevado/issues/81

NevadoJMSCacheManagerPeerProviderFactory, Ehcache uses this as its cacheManagerPeerProviderFactory


package com.bitsofinfo.ehcache.jms;

import java.util.Properties;

import javax.jms.ConnectionFactory;
import javax.jms.Queue;
import javax.jms.QueueConnection;
import javax.jms.Topic;
import javax.jms.TopicConnection;

import org.skyscreamer.nevado.jms.NevadoConnectionFactory;
import org.skyscreamer.nevado.jms.destination.NevadoQueue;
import org.skyscreamer.nevado.jms.destination.NevadoTopic;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

import net.sf.ehcache.CacheManager;
import net.sf.ehcache.distribution.CacheManagerPeerProvider;
import net.sf.ehcache.distribution.CacheManagerPeerProviderFactory;
import net.sf.ehcache.distribution.jms.AcknowledgementMode;
import net.sf.ehcache.distribution.jms.JMSCacheManagerPeerProvider;

public class NevadoJMSCacheManagerPeerProviderFactory extends CacheManagerPeerProviderFactory {

    @Override
    public CacheManagerPeerProvider createCachePeerProvider(CacheManager cacheManager, Properties props) {
        
        try {
        
            ApplicationContext context = new ClassPathXmlApplicationContext("/com/bitsofinfo/ehcache/jms/nevado.xml");
            
            NevadoConnectionFactory nevadoConnectionFactory = (NevadoConnectionFactory)context.getBean("connectionFactory");
            TopicConnection topicConnection = nevadoConnectionFactory.createTopicConnection();
            QueueConnection queueConnection = nevadoConnectionFactory.createQueueConnection();
            
            Topic nevadoTopic = (NevadoTopic)context.getBean("ehcacheJMSTopic");
            Queue nevadoQueue = (NevadoQueue)context.getBean("ehcacheJMSQueue");
            
            return new JMSCacheManagerPeerProvider(cacheManager,
                                                   topicConnection,
                                                   nevadoTopic,
                                                   queueConnection,
                                                   nevadoQueue,
                                                   AcknowledgementMode.AUTO_ACKNOWLEDGE,
                                                   true);
        } catch(Exception e) {
            e.printStackTrace();
            return null;
        }
    }

}

 

 

nevado.xml (NevadoJMSCacheManagerPeerProviderFactory boots this to init Nevado topic/queue @ AWS)


<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
           http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
          
           http://www.springframework.org/schema/context
           http://www.springframework.org/schema/context/spring-context-3.0.xsd">
           
      
    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
      <property name="locations">
        <list>
          <value>classpath:/com/bitsofinfo/ehcache/jms/aws.properties</value>
        </list>
      </property>
    </bean>
           
    <bean id="sqsConnectorFactory" class="org.skyscreamer.nevado.jms.connector.amazonaws.AmazonAwsSQSConnectorFactory" />
    
    <bean id="connectionFactory" class="org.skyscreamer.nevado.jms.NevadoConnectionFactory">
      <property name="sqsConnectorFactory" ref="sqsConnectorFactory" />
      <property name="awsAccessKey" value="${aws.accessKey}" />
      <property name="awsSecretKey" value="${aws.secretKey}" />
    </bean>
    
    <bean id="ehcacheJMSTopic" class="org.skyscreamer.nevado.jms.destination.NevadoTopic">
          <constructor-arg value="ehcacheJMSTopic" />
    </bean>
    
    <bean id="ehcacheJMSQueue" class="org.skyscreamer.nevado.jms.destination.NevadoQueue">
          <constructor-arg value="ehcacheJMSQueue" />
    </bean>

    
           
 </beans>

 

 

ehcache.xml

<ehcache>

   <diskStore path="user.home/ehcacheJMS"/>

    
    <cacheManagerPeerProviderFactory
       class="com.bitsofinfo.ehcache.jms.NevadoJMSCacheManagerPeerProviderFactory"
       properties=""
       propertySeparator="," />
       
       
     <cache name="testCache"
           maxElementsInMemory="1000"
           maxElementsOnDisk="2000"
           eternal="false"
           overflowToDisk="false"
           memoryStoreEvictionPolicy="LRU"
           transactionalMode="off">
           
           <cacheEventListenerFactory
                 class="net.sf.ehcache.distribution.jms.JMSCacheReplicatorFactory"
                 properties="replicateAsynchronously=true,
                              replicatePuts=true,
                              replicateUpdates=true,
                              replicateUpdatesViaCopy=true,
                              replicateRemovals=true,
                              asynchronousReplicationIntervalMillis=1000"
                  propertySeparator=","/>

      </cache>
    
       
</ehcache>

EhcacheNevadoJMSTest – little test harness program, run multiple instances of this w/ breakpoints to see ehcache utilize JMS(nevado/sns) to broadcast cache events


package com.bitsofinfo.ehcache.jms;

import net.sf.ehcache.Cache;
import net.sf.ehcache.CacheManager;
import net.sf.ehcache.Element;

import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class EhcacheNevadoJMSTest {

    public static void main(String[] args) throws Exception {
        
        ApplicationContext context = new ClassPathXmlApplicationContext("/com/bitsofinfo/ehcache/jms/bootstrap.xml");
        
        CacheManager cacheManager = (CacheManager)context.getBean("cacheManager");
        
        Cache testCache =cacheManager.getCache("testCache");

        Element key1 = testCache.get("key1");
        Element key2 = testCache.get("key2");
        key1 = testCache.get("key1");
        
        testCache.put(new Element("key1", "value1"));
        testCache.put(new Element("key2", "value2"));
        testCache.remove("key1");
    }

}

 

 

bootstrap.xml – used by the test harness


<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans-3.0.xsd


http://www.springframework.org/schema/context


http://www.springframework.org/schema/context/spring-context-3.0.xsd">

<bean id="cacheManager" class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean">
<property name="configLocation" value="classpath:/com/bitsofinfo/ehcache/jms/ehcache.xml"/>
</bean>

</beans>

 

 

 

 

 

 

 

 

 

About these ads

One Trackback

  1. […] post is a followup to what is now part one, of my research into using Ehcache, JMS, Nevado, AWS, SNS and SQS to permit cache events to be […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 28 other followers

%d bloggers like this: