In the recent past, a team I was working with was facing an architectural decision regarding what technology and deployment footprint to go with for a greenfield project.
Its been about five months now since this application has been in production.
The use-case in question was to present a suite of REST services to front a large set of “master data” dimensions for a data warehouse as well securing that data (record level ACLs). In addition to this, the security ACLs it would manage needed to be transformed and pushed downstream to various legacy systems.
With that general use-case in mind, some other items of note that were to be considered:
- The specific use case of “REST services facade for master data” was generic in nature, so the security model was to be agnostic of the specific data set being secured and have the capability of being applied across different data sets for different clients.
- Changes for a given service should be easy to fix and deploy and independent of one another with minimal interruption.
- The services need to scale easily and be deployed across several different data centers which are a mix of traditional bare-metal/ESX vm’s as well as in the cloud (azure/aws). Tight coupling to each DC should be minimized when possible.
- The services stack would potentially serve as the hub for orchestrating various other ETL related processes for the warehouse, so adding new “services” should be easy to integrate into the larger application.
- Given the sensitivity of the data, all traffic should be secured w/ TLS and REST apis locked down w/ OAuth2 client credentials based access.
Given the above requirements and much discussion we decided to go with a container based microservices architecture.
First off, this team already had significant experience w/ the traditional monolithic approach to applications and had already run into the many shortcomings of this architecture over the long term. As new features needed to be deployed, it was becoming more of a pain to add new “services” to the monolith as it required the entire stack to be redeployed which is disruptive. Given this new application would have a similar lifecycle (new services needing to be added over time) we wanted to try a different approach…. and who was the new kid on the block? “microservices”; and it was time to get one’s feet wet.
This shop was primarily focused on NodeJS, LAMP and Java stacks so after doing some research the decision was made to go with Spring Cloud as the base framework to build this new suite of services. If one does any reading on the topic of microservices, you will quickly see such architectures involve many moving parts: service discovery, configuration, calling tracing (i.e. think google dapper), load balancing etc.
Do you want to write these pattern implementations this all yourself? Probably not; I sure didn’t. So after evaluating the space at the time, Spring Cloud was the most robust solution for this and one of its biggest selling points is that it was based on many of the great frameworks that have come out of Netflix’s OSS project (Eureka, Hystrix and more..)
Lastly the decision to go w/ Docker was really a no brainer. The services would potentially need to be deployed and moved across various data centers. By using Docker DevOps would be able to have a footprint and deployment process that would be consistent regardless of what data center we would be pushing to. The only data center specific particulars our DevOps guys had to care about was, setting up the Docker infrastructure (i.e. think Docker hosts on VMs via Ansible coupling to DC specific host provisioning APIs) and the DC specific load balancers, who’s coupling to the application was just a few IP’s and ports (i.e. the IPs of the swarm nodes with exposed ports of our Zuul containers). Everything downstream from that was handled by Docker Swarm and the microservices framework itself (discovery, routing etc)
The acronym for this services backend ended up being CELL which stands for… well whatever you want it to stand for…. I guess think of it (the app) as an organism made up of various cells (services). CELL’s services are consumed by various applications that present nice user interfaces to end users.
The above diagram gives a high level breakdown of its footprint. Its broken up into several services:
Core services that all other app services utilize:
- cell-discovery: Netflix Eureka: Participating services both register on startup and use this to discover the cell-config service (to bootstrap themselves) plus discover any other peer level services they need to talk to.
- cell-config: spring-cloud-config: Git sourced application configuration (w/ encryption support). Each application connects to this on startup to configure itself.
- oauth2-provider: All services are configured w/ an OAuth2 client credentials compliant token generation endpoint to authenticate and get tokens that all peer services validate (acting as resource servers)
- tracing-service: zipkin: All services are instrumented w/ hooks that decorate all outbound http requests (and interpret them upon reception) with zipkin compliant tracing headers to collect call tracing metrics etc. Background threads send this data periodically to the tracing service.
- cell-event-bus: kafka and spring-cloud-stream: Certain services publish events that other services subscribe to to maintain local caches or react to logic events. This provides a bit looser coupling than direct service to service communication; leveraging Kafka gives us the ability to take advantage of such concepts of consumer groups for different processing requirements. (i.e. all or one)
- cell-router: Netflix zuul: Router instances provide a single point of access to all application services under a https://router/service-name/ facade (discovered via the discovery service). Upstream data center specific FQDN bound load balancers only need to know about the published ports for the Zuul routers on the swarm cluster to be able to access any application service that is available in CELL.
- cell-service-1-N: These represent domain specific application services that contain the actual business logic implementation invoked via external callers. Over time, more of these will be added to CELL and this is where the real modularity comes into play. We try to stick to the principle of one specific service per specific business logic use-case.
As noted above, one of the requirements for CELL was that participating services could have data they manage, gated by a generic security ACL system. To fulfill this requirement, one of those domain specific apps is the cell-security service.
The cell-security service leverages a common library that both cell-security servers and clients can leverage to fulfill both ends of the contract. The contract being defined via some general modeling (below) and standard server/client REST contracts that can easily be exposed in any new “service” via including the library and adding some spring @[secConfig] annotations in an app’s configuration classes.
- Securable: a securable is something that can have access to it gated by a SecurityService. Securables can be part of a chain to implement inheritance or any strategy one needs.
- Accessor: is something that can potentially access a securable
- ACL: Binds an Accessor to a Securable with a set of Permissions for a given context and optional expression to evaluate against the Securable
- SecurableLocator: given a securable‘s guid, can retrieve a Securable or a chain of Securables
- AccessorLocator: given a accessor‘s guid, can retrieve the Accessor
- AccessorLocatorRegistry: manages information about available AccessorLocators
- SecurableLocatorRegistry: manages information about available SecurableLocators
- ACLService: provides access to manage ACLs
- PrincipalService: provides access to manage Principals
- LocatorMetadataService: provides access to manage meta-data about Securable|Accessor Locators
- ACLExpressionEvaluator: evaluates ACL expressions against a Securable
- SecurityService: Checks access to a Securable for a requesting Accessor
The model above is expressed via standard REST contracts and interfaces in code, that are to be fulfilled by a combination of default implementations and those customized by individual application CELL services who wish to leverage the security framework. There are also a few re-usable cell-security persistence libraries we created to let services that leverage this to their persist security data (both authoritative and local consumer caches) across various databases (Mongo DB and or JPA etc). As well a another library to hook into streams of security events that flow through CELL’s Kakfa event bus.
Spring Cloud impressions
When I started using Spring Cloud (in the early days of the Brixton release), I developed a love – hate relationship with it. After a few initial early successes with a few simple prototypes I was extremely impressed with the discovery, configuration and abstract “service name” based way of access peer services (via feign clients bound to the discovery services)…. you could quickly see the advantageous to using these libraries to really build a true platform that could scale to N in several different ways and take care of a lot of the boilerplate “microservices” stuff for you.
That said, once we really got into the developing CELL we ended up having two development paths.
The first being one team working on creating a set of re-usable libraries for CELL applications to leverage and integrate into the CELL microservice ecosystem. This consisted of creating several abstractions that would bring together some of the required spring cloud libraries, pre-integrated via base configuration for CELL, and just make it easier to “drop-in” to a new CELL app without having to wade into the details of spring cloud too much and just let the service developer focus on their service. The amount of time on this part was about 70% of the development effort, heavily front loaded in the start of the project.
The second being the other team using the latter to actually build the business logic services, which was the whole point of this thing in the first place. This accounted for about 30% of the work in the beginning and today… about 80-90% of the work now that the base framework of CELL is established.
The hate part (well not true hate, but you know what I mean… friendly frustration) of this ended up being the amount of man hours spent in the start of the project dealing/learning spring-cloud. There is a tangible learning curve to be aware of. Working around bugs, finding issues in spring-cloud, both real ones or just working through perceived ones via misunderstandings due to the complexity of spring-cloud itself.
I’m not going to go into each specific issue here, however there were simply a lot of issues and time spent debugging spring cloud code trying to figure out why certain things failed or to learn how they behaved so we could customize and properly configure things. In the end most of the issues could be worked around or were not that hard to fix…. its just the time it took to figure out the underlying causation’s, produce a reproducible sample and then convey it to the spring-cloud developers to get help with. (The spring-cloud developers BTW are excellent and VERY responsive) kudos to them for that.
Lastly, taking each CELL artifact (jar) and getting it wrapped up in a Docker container was not an huge ordeal. In the deployed footprint, each CELL artifact is a separate Docker Swarm Service that is deployed on its own overlay network (separate one per CELL version). As stated previously, the CELL router (Zuul) is the only service necessary to be exposed on a published swarm port and then upstream datacenter load balancers can just point to that.
So would I recommend Spring-Cloud?
Yes. Spring Cloud at its heart is really an pretty impressive wrapper framework around a lot of other tools that are out there for microservices. It has a responsive and helpful community. (definitely leverage Gitter.im if you need help!) The project has matured considerably since I first used it and many of the issues I was dealing with are now fixed. Compared to writing all the necessary things to have a robust microservices ecosystem yourself….. I’ll take this framework any day.
Final note. I would NOT recommend using spring-data-rest. We used that on a few of the CELL application logic services and its main benefit of providing you a lot of CRUD REST services in a HATE-OS fashion…. its just not that easy to customize the behavior of, has a lot of bugs and just generally was a pain to work with. At the end of the day it would have just been easier to code our own suite of CRUD services instead of relying on it.
Everyone once in a while during the life cycle of any given piece of software comes that time where you have the opportunity to improve it in a major way….if that is, its lucky enough to still be in production.
One particular system I’ve been involved with is responsible for processing a lot of data and keeping that data in sync across many systems. For purposes of this little case study I’ve dumbed down the overall use-case, concept, architecture and implementation details to this simple idea. We need to synchronize data.
Something in the environment (i.e. a user or other process) makes a request for some operation to be done that generates a change operation against a “DataEntry”. This DataEntry is manipulated in the primary database and then the change needs to be synchronized numerous other systems to count. The changes could be described as “create DataEntry item number XYZ”, “Mutate DataEntry XYZ in fashion Z” or simply “Delete DataEntry item XYZ”.
Each target system where a DataEntry is to be synchronized is called a DataStore and involves its own complicated process of mutating our representation of a DataEntry into the target DataStore’s representation and the means to do it can vary wildly; i.e. web-service calls, RDBMS dml, nosql operations etc etc. Not to mention, as with any integration, each of these DataStore sync calls has the possibility being fast, very slow, not working at all, or experiencing random transient failures.
For most of its life the system functioned as follows, each DataEntry mutated in the system was placed in a queue, and then processed by a consumer node’s DataSyncProvider who’s responsibility is to determine all the DataStores to process the DataEntry in via interrogating a DataStoreLocator and then make sure it happens. It worked similar to the diagrams below (highly simplified!), and note the bottleneck.
Version 1 issues
Version 1 functioned fine for most of its life, however the biggest issues with is were simply its lack of efficiency and speed in synchronizing any given DataEntry across all of the DataStores it was applicable for. More often than not any given DataEntry mutation would result in dozens of target DataStores that it needed to be synchronized against. Due to the sequential processing of each DataStore, accommodating for retries, and waiting for the overall result….before moving on to the next one, this would result in a sizable delay until the mutation materialized in all target DataStores. (not to mention lack of good core utilization across the cluster). What did this mean? Well an opportunity for improvement.
Obviously, the choice here was to move to asynchronous parallel DataStore execution and decoupling from the main DataEntry mutation consumer thread(s)….. and there are many ways you could go about doing that. Fortunately the overall modeling of the synchronization engine enabled considerably flexibility in swapping out the implementation with a little refactoring. The key points being introducing the concept of a DataEntry logic execution engine; aptly named LogicExecutionEngine and adding a new implementation of our DataStoreLocator that could decouple any given DataStore’s location from any dependency on its actual residency within the local JVM.
Great. Now that the modeling is adjusted, what about implementation? For one, there was no interest it writing a multi-threaded execution engine, even though one could with the modeling in place; any implementation could have been be developed and plugged in. That said, after looking around for a good framework that provided location transparency, parallel execution management, clustering and good resiliency, it was decided that Akka, and moving to an Actor model for the new engine would be a good fit.
As shown above, the DataStores actually are now implemented via an ActorRef version which is then passed to the LogicExectionEngine who’s new Actor based implementation injects them into yet another Actor for the DataEntry logic processing awaiting a Future<Result>. This model increased overall execution time to completion by roughly 80% as everything now executed in parallel.
Another benefit was additional resiliency and distribution of load due to the location transparency of the actual DataStore itself. Utilizing Akka’s various Routers, such as in this case the ClusterRouterGroup Actor, we were able to further redistribute the processing of any given DataStore workload across the cluster and appropriately react as nodes came on and offline. See exploded view below.
Lastly, the diagram below shows how execution of these DataEntry tasks is now more evenly distributed across the entire set of available nodes in the cluster. All nodes can now be potentially involved in processing any DataEntry workload. Also by feeding dynamic configuration into the construction of each ClusterRouterGroup Actor the system could also fine tune the distribution and amount of Actors in the cluster that are available to process entries targeted at any given DataStore. This permits for custom down-scaling based on the limitations or load ceilings that any given downstream target DataStore may present. In other words it permits throttling of loads.
Overall my experience with Akka was positive. After working some of the bugs out, so far in production this solution has been quite stable and Akka’s clustering protocol quite stable. If you are considering moving to a more reactive design approach for the back end of a system, I highly recommend giving Akka a consideration.
Lastly, as always I highly recommend going with a pure interface oriented design in any system you build. In this use-case, this system’s entire platform itself, having been designed from the ground up using interfaces extensively and then plugging in different “providers” (i.e. things like Camel or Akka) for each aspect of implementation has proved out to be very important as it has evolved over time. This gives the system tremendous flexibility as it matures over time and additional longevity.
Recently I read Sam Newman’s “Building Microservices” , at ~280 pages its a fairly quick read. The reviews on this book overall are mixed and I can see where readers are coming from. By the title of this book one might expect some coverage of some of the microservices frameworks out there, concrete examples, maybe some actual code… but you won’t really find that here. Instead you will find a pretty good overview of various architectural approaches to modern application design in today’s world; covering general topics such a proper separation of concerns, unit-testing, continuous integration, automation, infrastructure management, service discovery, fault tolerance, high-availability and security etc.
In reality, none of the principles covered in this book are the exclusive domain of “microservice” application architectures, but rather can (and should be) applied to any application you are considering deploying; whether its a “monolithic” application or a suite of microservices interacting as parts of a larger functioning application.
In that right I think this book is definitely a good read and worth a look, if for nothing more than to ensure your team gets a refresher on good design principles and how they can be materialized with some of the newer frameworks and tool sets that have come out of our community in recent years. The material presented is sound.
I picked this book up a while back after looking for a book on Java EE patterns. The mainstream standard seemed to be the Core J2EE Patterns book, but the more I looked at it it just seemed outdated. So when I found Adam’s book just the title looked practical “Rethinking Best Practices”. So in short I’ve finally read it and was quite impressed. This is a great book. The author basically goes through the standard patterns and tackles each one by explaining its overall objective, the forces at play, a bit how it works, THEN… the big part how to re-think it in the context of Java EE ejb3/3.1. For example the author does a good job explaining how DAO’s are now simply replaced by JPA’s EntityManager and everyone should really re-consider if they still need this abstraction layer in green field projects. Although he does admit that they can still serve a purpose as a place to consolidate boilerplate, common EntityManager related code. Regardless, pattern by pattern, the author does a great job giving real world examples of how these standard patterns can be modified or adapted to the ejb3 realm. (He also covers which J2EE patterns can now be retired). One of my favorite parts of the book which gave me a great glimpse on how to tackle an immediate problem I am faced with was “Dependency Injection Extender” pattern. This will come in use for me as we have a Spring codebase which we will want to utilize in a JBoss environment. I’d like to use @Inject and JSR330 but our container (JBoss 5) does not yet support that. So by using this pattern with interceptors I think we will be able to annotate our beans, test them outside of JBoss, yet still wire everything up properly using this idea presented in the book.
Downsides to the book? Some of the headings, intros to subsections are not bolded in the text when it appears they should have been.
Overall, 5 out of 5 for this book. I recommend it! You can tell this was written by someone with a ton of experience under their belt.
This is a book review for “97 Things Every Software Architect Should Know” by O’Reilly with dozens of contributors.
This book is a quick read at roughly 200 pages and is targeted towards those folks who find themselves in the role of the “software architect”. Many of the contributors will be names you recognize such as Mike Nygard, Neal Ford and many others, some of which get more “booktime” than others with multiple blurbs each.
Each small contribution is no more than a page and a half as each author shares some personal experience, lesson learned or rule of thumb when it comes to being a architect. This may not be a book you sit down and read all at once, but maybe a few articles a day and move through it that way. I also think I might open it up again right before I start a brand new large project as this would be a good refresher on many concepts and methodologies which can help you get your mind in the right place before starting something new.
This book is not super technical in nature and is recommended for advanced developers or anyone who considers themselves to be an architect.
This is a review of “Interface Oriented Design” by Ken Pugh.
I don’t have a ton to say about this book, it was a fairly quick read at about 200 pages. If you have a lot of experience with OO, the concepts presented within it are nothing that startling, however they may serve the reader as a nice refresher or presentation of important principles in a “interface” oriented and focused approach. I think if anything, one handy thing that the author presented were the “laws” of any interface:
a) “Do what its methods say it does” : this is obviously important as how many times have you seen methods in a program that do what their name implies, but… sometimes add on a bit more undocumented functionality.
b) “Do no harm”: the implementation to should avoid affecting other modules or consuming resources that it does not explicitly declare it will etc.
c) “Provide meaningful errors”: I think this is often missing. Too many times I’ve called some method to get a horrible low-level exception thrown back at me, or a custom one with some un-intelligible error code. Rarely wil you get back something meaningful, like the author suggests which contains not only the problem, but why it occurred and how you can possibly avoid it on the next call.
Overall I found chapters 1-7 to be the most informative, while chapters 8-10 went through some examples which focused on the principles covered in 1-7 in working examples. Chapter 11 covers the standard GoF patterns but from a “interface” based perspective.
Overall, I would say that if you are pretty versed in OO you could get a way with skipping this book. Otherwise if you are intermediate or new to OO and the proper use of interfaces as opposed to constantly using inheritance, this might be a good book for you.
Lastly I will say that after reading this book, when faced with new design decisions, I found myself focusing much, much more on pure interface based designs and minimizing inheritance except where absolutely necessary.