Lets migrate everything to containers. Everything. The app infrastructure, the apps themselves, deployment workflows, DevOps procedures… everything.
Recently started winding down one of those “on and off again” projects with long term goals but was constantly susceptible to the day to day distractions and diversions typical for any team managing a ton of different apps.
Those goals were as follows:
- Re-architect the entire application infrastructure footprint to make it more datacenter
agnostic while in turn providing a consistent DevOps management experience.
- Bring consistency, flexibility and enable rapid change for the application deployment and management process.
Sounds straight forward…. so what was originally in place?
For years this group developed, released and managed dozens of applications coded in various languages and frameworks residing in general topology as follows:
Sitting behind various network layer firewalls and security devices; the proxy tier was essentially made up of a large farm of HTTP(s) reverse proxies responsible for:
- Optional front side SSL negotiation and termination (if not already handled upstream)
- Web application firewalls (WAF)
- Single sign on session management (SSO)
- Egress to applications
Physically this was implemented as a large farm of various HTTP servers; each of which running the same configuration. Configuration being defined as a separate virtual host for every possible site it was proxying traffic for. Each one of these configuration’s, per proxied site, consisted of specific static configurations for each of the responsibilities listed above (SSL, WAF, SSO).
Much of this static configuration was managed in source control, edited and pushed out in bulk via various DevOps toolsets (Puppet, bulk SCP routines, custom scripts etc) to all participating proxy hosts. A certain percentage of this configuration was literally duplicated, while those with minor customizations still tended to follow repetitive patterns of configuration. None of the configuration was dynamically generated. Due to the nature of the underlying proxy’s configuration format, this ended up being a quagmire of duplicated blocks, stanzas, patterns, in-consistent use of includes etc. It was also somewhat laborious to do simple things like simply change certificate, add support for a new site or just another alias FQDN for an existing one. The system and process created a blurred set of responsibilities between the development and DevOps as over time the proxy layer evolved into serving some aspects of what could be considered application level logic via url rewrites, content transformations and sometimes a mechanism for the conveyance of application specific configuration values etc.
Behind this set of proxies were the actual application servers running a wide array of frameworks and languages. Each proxy server had numerous different “egress pools” defined via more configuration files that pointed to a list of app servers whereby actual requests were proxied to. What defined a egress pool could be generic (i.e. “pool-ruby-apps”) or very application specific (i.e. “pool-[appname]”).
The app tier was the set of hosts responsible for actually serving up the end applications that the proxy tier was sending traffic to.
These could be split up into 2 categories:
- Apps on a fat HTTP server stacks: i.e. applications coupled to execute within traditional HTTP server processes via language specific modules/plugins.
- Everything else: i.e apps, frameworks and stacks that provide their own embedded http(s) network stacks.
For each unique combination of the above there was a unique farm of servers capable of running each classification of application. Some farms were shared execution environments (i.e. traditional HTTP server module/plugin embedded apps) while others were dedicated to a single app. For HTTP server coupled environments there again was a large set of static configuration living in source control, containing the same kinds of issues as the proxy tier (duplicated HTTP server related stanzas/patterns, repetitive config patterns, in-consistent use of includes etc). What was even more complicated is that this layer was where the lines of responsibility between Devs and DevOps blurred in regards to configuration.
DevOps responsibilities included things like adding new virtual hosts or modifying things like supported server names, aliases etc, or just tweaking low level HTTP server options; while over time, Dev started using the HTTP server’s support for variables as a means to convey certain types of application level config variables/flags, not to mention rewrites, path mutations etc. In short, over time the lines blurred and application level configs bled into HTTP server configs simply because it was easy to do so and often was a quick fix when compared with facing the reality of harder to refactor architectural issues.
As time progressed, costs emerged as the number one driver and that led to a situation where the topology ended up being difficult to upgrade… especially on shared stack environments where the funds were simply not there to add new servers as upgrade/migration targets. This then led to reduced patching intervals and very tedious and laborious efforts when upgrades were scheduled. For example, in shared footprints it was quite painful if only a single application needed a library upgrade. Releases had to be very careful orchestrated, with scheduled downtime across different application owners etc.
Overall, it was not an ideal situation.
Anticipating change: V1 infrastructure architecture footprint
Starting in 2015 and continuing on and off in 2016 early work began exploring containerization as a means to tackle the issues presented in the architecture above. At that time the entire world of containerization was still relatively new and the world of container orchestration was even more grey, vaguely defined and immature. Hence a decision was made to explore an initial architecture based around a three tiered vanilla Docker (swarm legacy standalone) architecture along with Registrator and Consul for service discovery.
- “Administrative” Tier: N Docker hosts running core services such as a private Docker Registry, Consul (config management, service discovery), Vault (configuration, secrets), Registrator and Docker Swarm (standalone) managers
- N “App” Tiers: separate swarm standalone “clusters” of Docker hosts each running as “swarm workers” slaved to respective swarm managers on the Administrative tier. Each node had local Consul agent and Registrator containers. The goal was different “App” cluster swarms for different workloads
- N “Load Balancer” tiers: N Docker Hosts running NGNIX containers along with Registrator and Consul agent containers. Each NGINX container interrogated Consul via consul-template to dynamically populate its proxy target App Tier service backends from Consul’s service discovery catalog.
The general workflow was as follows
- App containers would be deployed to a target App Tier swarm (there could be many swarms)
- Upon startup/shutdown of each app container (via legacy standalone swarm), each Docker host’s Registrator container would register the app container under an appropriate “service name” within Consul along with its published ports and other meta-data if need be. Containers published ports would be assigned on any random available high port on each swarm member’s Docker daemon; there were no “fixed” published ports.
- Optionally, each app container could leverage consul-template to connect to an appropriate Consul and/or Vault service to pull configuration and secrets as well as interrogate Consul to discover other named peer services.
- In order for app containers on the app tier to actually receive traffic, they needed one or more corresponding NGINX load-balancing containers deployed on the Load balancing tier.
- The NGINX load balancer containers were purposely dumb, their only responsibilities were optional SSL termination and proxying traffic (no rewrites or other logic). On startup one of their configuration parameters was the “name” of the corresponding service in Consul they would proxy traffic to. Using consul-template, the appropriate configurations could be rendered (and updated real-time) to send traffic to the right backend app containers running on the App Tier. This could dynamically react accordingly as the backend app containers came up/down changed due to consul-template “watches” against the Consul service catalog.
- The NGINX load balancer containers DID have FIXED published ports for each service. This was so upstream load-balancers (hardware or cloud balancers) could point to the correct backends for a given DNS name bound for that service to the datacenter/cloud specific LB device.
V1 app remediation
Also at this time significant work had to be done to take the legacy “proxy tier” and “app tier” components and figure out how to effectively reproduce those functions in a container image architecture that would permit both migration to the new V1 platform as well as immediate customization by application owners. The app remediation portion of the new design was the most significant and time consuming, even more so than the underlying infrastructure part described in the previous section.
The main goals for this task were as follows:
- Completely eliminate as much hard-wired configuration as possible.
- Ensure that the configuration that needs to be customized is completely stripped away from binary image artifacts and move to a model whereby configs are dynamically sourced at boot/runtime. (for both apps and L7 proxies)
- Permit each individual app and or layer 7 proxy to diverge from others, permit upgrading different L7 layers for different apps to be upgraded independently from one another.
- To permit maximum flexibility, eliminate any “shared” conduits of traffic. Keep a one-to-one relationship between L7 proxies and the actual app behind it. This leads to more containers, but they are all small and can be sized appropriately per individual workload rather than the legacy way of one size fits everything…
- Support a very flexible container deployment topology. i.e. a single container could run all L7 proxies, or tier it out into any combination thereof.
The result of this work ended up looking similar to the following:
The latter basically provided a suite of core image layers that could be combined into any combination to produce derivative artifacts that were completely driven by configuration sourced externally via tooling like consul-template.
- The “http-core” image layers (several variants) was responsible for a base http server, ssl support, core logging features, basic http server related configs etc.
- The “waf-core” built on top of the http-core layer adding customizable WAF functionality
- The “sso-core” built on top of either of the previous two layers installing SSO authn/z functionality
- From there any derivative layers could be based FROM the core layers and could optionally decorate their own configuration to customize any of the base image layers which would automatically look for customized includes based on known conventions. Essentially all base layers had various hook points for customization.
Each “layer” of core functionality operated on a simple contract:
“Provide a single virtual host that listens on a configurable port and egresses to a configurable URL.”
Given this setup, different image stacks could be built and resulting artifacts could be deployed in any combination on the network and configured to talk to one another to produce any physical L7 topology one desired; from single stack (all proxied within a single container), to N-legged setups across a container orchestrator’s network. This topology was uniquely separate from the underlying physical infrastructure it might run within which could be setup to further segment flows and limit access to/from certain source/destinations.
The above design turned out to be a game changer in the way apps and their supporting proxies are built, managed, deployed and configured. This opened up a world of new possibilities with regards to consistent methods of project management, breaking up legacy monoliths of infrastructure, externalized configuration and new approaches to consistent deployment regardless of the actual artifact in question.
but….V1 was never to be…
The above v1 architecture worked pretty well during early R&D and testing however it never made it into production as other greenfield business application development initiatives coupled with a data center migration became the priority. Attempting to do all of this tied together with a major containerization project would be too much to tackle. The containerization project was put on hold in favor business priorities.
Adios V1….hola V2
After the other priorities were finished and settling in; throughout 2016 and into this year many things have evolved in the world of container technology.
The containerization project was back in focus. Fortunately for us several of the components in the V1 infrastructure architecture by this point have become superseded by newer projects or have had underlying support baked in natively.
Case in point: by this time Docker’s “swarm mode” in 1.12+ had become available so we revisited the entire design. This led to re-visiting the original plan by doing some new testing and eventually changing some key parts of the V1 architecture, leading to V2. Fortunately all of the work invested in the initial app container image architecture and administrative backends were all still applicable.
It essentially boiled down to using a more advanced orchestrator while eliminating some hobbled together components that were now provided “out of the box” around service discovery and proxying to service backends. The Registrator project also appeared to be about dead, so eliminating that was something desired. The new kid on the the block, Traefik, was about to change a lot for us.
The biggest changes were as follows:
- Eliminate using swarm standalone legacy: (i.e. no more swarm manager/agent containers on any tier)
- App tiers simplified and just became Docker swarms using swarm mode
- The custom load-balancer tier could also now be eliminated and replaced with Traefik services running on each swarm directly. This would save managing a lot of extra moving parts.
- The use of Registrator was no longer necessary on the app tier as the combination of Docker Swarm combined with Traefik provided all the functionality the previous architecture (registrator, consul services and LB tier) offered but in V2 was now an automated fashion using tooling we neither have to maintain or configure. Via “labeling” a lot of magic just happened. Gone was the need to do a lot of our own “orchestration” via a slew of other containers.
Surprisingly the changes described above were quite easy from an architecture standpoint and greatly simplified things. The power of Traefik to fully automate everything the prior architecture was providing via the combination Registrator, Consul Services and NGINX lb containers is impressive. In short this change is actually speeding up our time to start moving forward again and focus on converting apps and preparing the pathway to production which is the next major goal.
I’ll do another post that as the experience progresses.
April 2018: Its been a few months and V2 has come along quite nicely. I’ve added an article covering the resulting architecture that is turning out to work well as things migrate into production. You can read more at: https://bitsofinfo.wordpress.com/2018/04/18/architecture-docker-swarm-traefik/
Whats next? Who knows, probably Kubernetes given the way the orchestrator landscape is settling in.