Orchestration of infrastructure at Prozeta relies on the “role and profile” principle, using an orchestration stack based on Puppet, Foreman and Hiera.
Puppet and Foreman
Our orchestrated infrastructure is based primarily on Foreman and its surrounding ecosystem – Puppet, PuppetDB, Hiera and r10k.
We incorporate all of these components into our home-grown project called Mother, referring to a higher level “orchestration of orchestrator”. This allows us to easily create isolated orchestrators for cloud environments just by creating a Hiera configuration and spawning a VM.
We utilize the benefits of the well-known Discovery plugin for Foreman, which manages our physical machines’ provisioning and helps us configure essential parts of the installed OS (networking, partitions, etc.).
Here comes the most powerful factor: when you orchestrate everything using Foreman tools, you get all your data in one place: everything to do with HW and system (facts), roles & profiles (hostgroup) and system configuration state (Puppet reports).
Roles & Profiles
We have one massive profile Puppet module named BlackStack, which orchestrates everything about OpenStack that we need. This module uses many public puppet modules (Puppet Forge) which are resolved as dependencies by r10k. The profile class is then applied to all machines of an isolated stack (which we call a “section”) held in Foreman’s hostgroup, which is later divided into sub-hostgroups by a machine`s (or ring of machines’) role, i.e.
- controller node
- monitoring node
- graphing node
- compute node
- storage node, etc.
Grouping nodes by roles opens up the possibility of auto-grouping and auto-discovery of clusters, so horizontal scaling is just a matter of adding a provisioning machine to the right host group.
Then we fetch a list of machines with the same role from PuppetDB (with help from the puppetdbquery module) and re-configure clusters with the new array of hosts. When you know system facts about all the machines in a stack, you can then easily feed them anywhere needed – e.g. your monitoring can automatically start monitoring new IPMI addresses, create every-to-every ICMP checks, etc.
Each section has its Hiera manifest, paired by hostgroup parameters evaluated in Puppet’s ENC. Any exceptions can be handled in Hiera’s host config file, so this solution is well-defined, self documented, yet flexible.
Environments
For needs of development and staging, we use Foreman’s (and Puppet’s) environments feature. Using r10k software, we can create different Puppet environments (using git branches and a different Puppet file). All deployments are instigated by drone-ci and triggered by git hooks, so all pre-deploy errors like broken dependencies or syntax errors are caught before they land on servers.
Foreman API
The power and elegance of all the afore-mentioned tricks together can be seen when you use Foreman’s hammer-cli and its extensions. With these, all that’s required to deploy a stack is one Hiera config file and a hammer-cli powered script that will deploy right-sized machines into the appropriate hostgroups.