Hazelcast discovery with Consul

I’ve used Hazelcast for years and have generally relied upon the availability of multicast for Hazelcast cluster discovery and formation (within a single data-center). Recently was faced with two things, expand the footprint into a non-multicast enabled data-center and secondly pre-prep the service for containerization where nodes will come and go as scaling policies dictate it…. hardwired Hazelcast clustering via an XML configuration and/or reliance on multicast is a no-go.

With Hazelcast 3.6, they now support a pluggable implementation for a cluster discovery mechanism called the Discovery SPI. (Discovery Strategy) Perfect timing, given we are already playing with Consul as part of our Docker container strategy, this was an opportunity to let our application’s native clustering mechanism (coded on top of Hazelcast) to leverage Consul as well as discover/remove peers both within, and potentially across data-centers.

So I coded up hazelcast-consul-discovery-spi available on GitHub.

diag.png

This works with Hazelcast 3.6-EA+ and Consul to provide automatic registration of your hazelcast nodes as Consul services (without having to run a local Consul agent) and automatic peer discovery of the Hazelcast cluster.

Note that the automatic registration of each hazelcast instance as a Consul service is OPTIONAL if you already have Consul agents running that define your Hazelcast service nodes. I added that in simply because I think it will be convenient for folks, especially when containerizing a Hazelcast enabled app (such as via Docker) where the less “dependencies” like a Consul agent available on the host, or in the container (or another container).. the better. You can totally embedded this functionality with this discovery strategy SPI.

I hope others find this helpful, and please leave your feedback, pull-requests or issues on the project!

NOTE, if you are running your app in Docker you have a separate issue where you need to determine your own externally accessible IP/PORT that the docker host has mapped for you on 5701… well how can you determine that so that you can publish the correct IP/PORT info to Consul? Check out: https://github.com/bitsofinfo/docker-discovery-registrator-consul

NOTE! Interested in etcd? There is a separate project which is built around etcd for your discovery strategy located here: https://github.com/bitsofinfo/hazelcast-etcd-discovery-spi

 

Advertisements

4 comments

  1. Peter Heitman

    This looks like exactly what I need – but I’m having one key problem. Our application is Spring-based and I want to make this network strategy conditional on being in a deployment using consul. The spring xsd does not have the “enabled” attribute for the discovery strategy, unlike the other three options for the network join. Is there any examples that describe how to support conditionally enabling the discovery strategy? Possibly extending the class to look for an enabled property?

  2. Andreas Lindfalk

    What I did to skip the XML in a Spring Boot app:

    @Bean
    @Profile(AppProfile.NORMAL)
    public Config hzConfig() throws Exception {

    Config config = new Config();

    // Use Consul for discovery instead of multicast with the help of this:
    // https://github.com/bitsofinfo/hazelcast-consul-discovery-spi

    config.setProperty(“hazelcast.discovery.enabled”, “true”);
    config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);

    DiscoveryStrategyConfig discoveryStrategyConfig = new DiscoveryStrategyConfig(new ConsulDiscoveryStrategyFactory());
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_HOST.key(), this.consulHost);
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_PORT.key(), this.consulPort);
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_SERVICE_TAGS.key(), “”);
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_SERVICE_NAME.key(), “hz-” + this.appName);
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_HEALTHY_ONLY.key(), true);
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_DISCOVERY_DELAY_MS.key(), “3000”); // TODO Fix
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_REGISTRATOR.key(), LocalDiscoveryNodeRegistrator.class.getName());

    ObjectNode jsonRegistratorConfig = JsonNodeFactory.instance.objectNode();

    jsonRegistratorConfig.put(“preferPublicAddress”, false);
    jsonRegistratorConfig.put(“healthCheckProvider”, HttpHealthCheckBuilder.class.getName());
    jsonRegistratorConfig.put(“healthCheckHttp”, String.format(“http://#MYIP:%s%s”, this.serverPort, this.healthCheckPath));
    jsonRegistratorConfig.put(“healthCheckHttpIntervalSeconds”, 3);

    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_REGISTRATOR_CONFIG.key(), jsonRegistratorConfig.toString());

    config.getNetworkConfig().getJoin().getDiscoveryConfig().getDiscoveryStrategyConfigs().add(discoveryStrategyConfig);

    return config;
    }

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s