Puppet at CSAIL

For the past several years, we have used a confusing and ill-specified mixture of FAI, the automatic Debian/Ubuntu installer, and cfengine to manage our Linux systems. Puppet is a newer, more flexible configuration-management system, and we are looking to replace the old FAI/cfengine system with Puppet as we replace systems. We are also trying to bring our FreeBSD systems under configuration management, to reduce the likelihood of bus-fault failures.

This page will summarize the various things we've done with Puppet and some of the pitfalls we've run into, and also include links to the Puppet modules we've either written or rewritten to meet our needs. Where practical, we are providing a clonable Git repository so that others can easily take advantage of what we've done; we welcome input from the Puppet community.

General issues

We've found that many of the PuppetForge modules are rather less useful than they first appear ("of course it's cross-platform -- it works on both Fedora and Ubuntu"). This requires us to keep our own private repositories for many of these modules rather than being able to reference the official public source. Whether the repository is local or remote, Git submodules are a real pain (albeit less painful than the other ways of doing the same thing in Git) -- that's particularly true when we need to support read-write access by administrators and read-only access by the Puppet servers to these shared repositories using the same URL across all systems (which do not all implement a shared filesystem).

Hiera

Our Hiera hierarchy is organized as follows:
  1. $::fqdn
  2. $::group
  3. $::role
  4. $::osfamily
  5. common

group and role are custom facts defined in /etc/facter/facts.d at installation time, and are used to identify the research group (Vision, Computer Architecture, Theory of Computation, etc.) and function (workstation, compute server, NFS server, etc.) for each system. common includes site-wide defaults such as IP addresses of management workstations, system administrator mailing-lists (who gets root@ mail) and similar variables that either don't vary at all across CSAIL or vary only across a limited subset of CSAIL machines. Because some of the configuration is either private or security-sensitive, we are not sharing the contents of our Hiera data files.

We would like to encourage all Puppet sites to implement a few standard Hiera variables; adopting a common practice would greatly improve the default out-of-the-box usability of many Puppet modules, and the modules that we are distributing here often depend on them. Currently, these are:

  • management_stations: An array of IP addresses (currently v4-only) which should be allowed access for management services like ntpdc, ntpq, munin, and nrpe2. Commonly these will be the systems you actually use for monitoring services, but in our case we also include sysadmin workstations.
  • local_netblocks: An array of IPv4 netblocks which are considered "local" to the management domain. In our implementation, we list the netblocks that we manage.
  • local_netblocks_v6: Likewise, but for IPv6.
  • campus_netblocks and campus_netblocks_v6: Arrays of netblocks which are considered "local-ish"; this should include both the "local" netblocks and also other networks that should be considered "inside" or "semi-inside". We put the whole MIT campus in these lists, which is much broader than just what we administer.

Modules we have implemented

  • csail: contains all of our procedural-style configuration (stuff that can't easily be represented in Hiera)
  • freebsd: miscellany for managing FreeBSD systems; includes a freebsd::ipfw class (we'll probably do the same thing for Linux if the need arises)
  • krb5: configure Kerberos
  • lldp: manages either openlldp or lldpd, depending on the platform
  • nfs: configures NFS servers (only) (on FreeBSD only) (so far)
  • ntp: configuration for NTP servers (the PuppetForge ntp module was not usable for servers)
  • openafs
  • smartd: configured smartd; includes some useful facts for LSI MegaRAID?/MegaRAID SAS/Dell PERC5/PERC6 controllers, but only lightly tested
  • sshguard: manages versions of sshguard recent enough to tail log files directly
(Links are git cloneable and also browsable using gitweb. Note that our modules require the use of a pkgng Puppet package provider and appropriate pre-compiled packages on FreeBSD.)

Modules from elsewhere that we have hacked on

  • monit: ported to FreeBSD and added support for automatic instantiation of service monitors
  • munin: ported to FreeBSD and added support for automatic instantiation of plugins
  • shell_config (taken from Puppet Wiki): a defined type which uses Augeas's Shellvars lens to manipulate files that look like shell variable assignments. We extracted this from our freebsd module when it became clear that Debuntu systems could make use of it as well, but we haven't converted everything over to this style yet. On FreeBSD we also use it for /boot/loader.conf, which requires a small tweak to the Augeas package in order to function properly. (Shellvars.lns doesn't want dots in key names, which is reasonable since the actual shell doesn't allow that, but the kernel environment variables in loader.conf require them.)

Modules that we were able to adopt without significant changes

  • OpenStack-related modules (glance, horizon, keystone, openstack, rabbitmq, swift)
  • apt
  • concat
  • razor (but we decided not to use Razor)
  • stdlib
  • sysctl
Topic revision: 07 Dec 2012, GarrettWollman
 

MIT Computer Science and Artificial Intelligence Laboratory

 

  • About CSAIL
  • Research
  • News + Events
  • Resources
  • People

This site is powered by Foswiki MIT: Massachusetts Institute of Technology