I’ve recently been working on a project where a set of application runtime configuration was stored in Consul by using Consul’s key-value store functionality. The application could dynamically materialize its configuration at runtime by using consul-template which is another handy tool by Hashicorp. Consul-template permits you “watch” a Consul KV space and automatically render Golang templates as key-value pairs change. The Golang templates references key-value “paths” in Consul and you can render your configuration once prior to application bootstrap or let it run as a daemon and re-render config changes as the KVs change in Consul via watches.
When using such a strategy to materialize configuration dynamically from Consul, its usually wise to have your key-value pairs under version control before it gets into Consul (which is your means of KV distribution). Without some kind of versioning you can get yourself into trouble.
So how to do this? Well there are numerous tools to help you with this, one of the first ones you might come across is git2consul which is a pretty good tool you can use (run on demand or as a cron job) that will auto sync the contents of a Git repository that contains a directory structure of files who’s contents define the value for the key (filename). git2consul (and its numerous forks) support numerous config options for branches/tags, filters for what to sync, how to map it to Consul KV spaces etc. git2consul works pretty good for simple (limited repository setups), however in my experience, if you are syncing dozens of different repositories and each repository has a high number of tags…. it doesn’t scale very well and unfortunately these issues are not being addressed as the project is dead for all practical purposes. I’m not knocking the tool, its a fantastic one, but it has its limitations and is not getting any PRs merged.
That said, there are numerous forks out there and similar tools:
- https://github.com/DataDog/git2consul
- https://github.com/Cimpress-MCP/go-git2consul
- https://github.com/KohlsTechnology/git2consul-go
- https://github.com/joeatbayes/file2consul
In light of the situation, the team could no longer reliably use the git2consul tool as they were maintaining their own fork with custom hacked fixes and it became unreliable. Secondly I also didn’t want to use one of the forks as they were simply too complicated for the refactored uses-case. Lastly as part of a larger CI/CD project with this team, I really wanted a much lighter weight tool that could effectively do the same thing and not require a custom configuration file and be invoked on demand as part of a CI build process.
The CI build flow would work like this:
- Clone the Git repo tag needing a build
- Test, build/push an Docker image
- Push the project’s config KV directory tree to a target Consul KV tree scoped by version/tag.
I had no need for a ton of custom configuration options, so I came up w/ a small tool for this called files-to-consul-kv and it works as follows by leveraging the Consul KV transactions API.
Its very simple:
cd mykvs/
$ find . -print
.
./sub
./sub/key2
./key1
$ cat key1
val1
$ cat sub/key2
val2You could use fs2consulkv.py to set all these in Consul under some root path:
docker run -i -v `pwd`/mykvs:/kvsource \ bitsofinfo/files-to-consul-kv fs2consulkv.py \ --fs-kv-path /kvsource \ --consul-url https://[consul-fqdn][:port] \ --consul-acl-token xxxxxxx \ --consul-data-center optional-dc \ --consul-kv-root some/root/path/
Would result in your KVs in consul at:
https://[consul-fqdn][:port]/ui/mydc/kv/some/root/path/key1 = val1
https://[consul-fqdn][:port]/ui/mydc/kv/some/root/path/sub/key2 = val2One and done, on demand. Hopefully you might find this useful:
bits.of.info 