Hazelcast discovery with Consul

Posted on by bitsofinfo

I’ve used Hazelcast for years and have generally relied upon the availability of multicast for Hazelcast cluster discovery and formation (within a single data-center). Recently was faced with two things, expand the footprint into a non-multicast enabled data-center and secondly pre-prep the service for containerization where nodes will come and go as scaling policies dictate it…. hardwired Hazelcast clustering via an XML configuration and/or reliance on multicast is a no-go.

With Hazelcast 3.6, they now support a pluggable implementation for a cluster discovery mechanism called the Discovery SPI. (Discovery Strategy) Perfect timing, given we are already playing with Consul as part of our Docker container strategy, this was an opportunity to let our application’s native clustering mechanism (coded on top of Hazelcast) to leverage Consul as well as discover/remove peers both within, and potentially across data-centers.

So I coded up hazelcast-consul-discovery-spi available on GitHub.

diag.png

This works with Hazelcast 3.6-EA+ and Consul to provide automatic registration of your hazelcast nodes as Consul services (without having to run a local Consul agent) and automatic peer discovery of the Hazelcast cluster.

Note that the automatic registration of each hazelcast instance as a Consul service is OPTIONAL if you already have Consul agents running that define your Hazelcast service nodes. I added that in simply because I think it will be convenient for folks, especially when containerizing a Hazelcast enabled app (such as via Docker) where the less “dependencies” like a Consul agent available on the host, or in the container (or another container).. the better. You can totally embedded this functionality with this discovery strategy SPI.

I hope others find this helpful, and please leave your feedback, pull-requests or issues on the project!

NOTE, if you are running your app in Docker you have a separate issue where you need to determine your own externally accessible IP/PORT that the docker host has mapped for you on 5701… well how can you determine that so that you can publish the correct IP/PORT info to Consul? Check out: https://github.com/bitsofinfo/docker-discovery-registrator-consul

NOTE! Interested in etcd? There is a separate project which is built around etcd for your discovery strategy located here: https://github.com/bitsofinfo/hazelcast-etcd-discovery-spi

 

Published by bitsofinfo

6 thoughts on “Hazelcast discovery with Consul”

  1. This looks like exactly what I need – but I’m having one key problem. Our application is Spring-based and I want to make this network strategy conditional on being in a deployment using consul. The spring xsd does not have the “enabled” attribute for the discovery strategy, unlike the other three options for the network join. Is there any examples that describe how to support conditionally enabling the discovery strategy? Possibly extending the class to look for an enabled property?

    Reply

  2. What I did to skip the XML in a Spring Boot app:

    @Bean
    @Profile(AppProfile.NORMAL)
    public Config hzConfig() throws Exception {

    Config config = new Config();

    // Use Consul for discovery instead of multicast with the help of this:
    // https://github.com/bitsofinfo/hazelcast-consul-discovery-spi

    config.setProperty(“hazelcast.discovery.enabled”, “true”);
    config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);

    DiscoveryStrategyConfig discoveryStrategyConfig = new DiscoveryStrategyConfig(new ConsulDiscoveryStrategyFactory());
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_HOST.key(), this.consulHost);
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_PORT.key(), this.consulPort);
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_SERVICE_TAGS.key(), “”);
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_SERVICE_NAME.key(), “hz-” + this.appName);
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_HEALTHY_ONLY.key(), true);
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_DISCOVERY_DELAY_MS.key(), “3000”); // TODO Fix
    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_REGISTRATOR.key(), LocalDiscoveryNodeRegistrator.class.getName());

    ObjectNode jsonRegistratorConfig = JsonNodeFactory.instance.objectNode();

    jsonRegistratorConfig.put(“preferPublicAddress”, false);
    jsonRegistratorConfig.put(“healthCheckProvider”, HttpHealthCheckBuilder.class.getName());
    jsonRegistratorConfig.put(“healthCheckHttp”, String.format(“http://#MYIP:%s%s”, this.serverPort, this.healthCheckPath));
    jsonRegistratorConfig.put(“healthCheckHttpIntervalSeconds”, 3);

    discoveryStrategyConfig.addProperty(ConsulDiscoveryConfiguration.CONSUL_REGISTRATOR_CONFIG.key(), jsonRegistratorConfig.toString());

    config.getNetworkConfig().getJoin().getDiscoveryConfig().getDiscoveryStrategyConfigs().add(discoveryStrategyConfig);

    return config;
    }

    Reply

  3. Quite useful…. Except it is not considering tags when discovering services registered with consul.
    Take for example multiple environments registered under same service name when differentiated by tags. As of now it fetches all host names, irrespective of tags. Filtering by tags should be considered too.
    Do you have any plans to incorporate tags in query options when fetching hosts from consul client?

    Reply

    1. There are no plans to but if you’d like to add that feature feel free to submit a pull request on the github project or create an issue there.

      Reply

Leave a comment