Normally, when we build libraries in our programming languages, we’re benefiting from a lot of existing design work. The language itself comes with many abstraction mechanisms to make our work useful.

If we write a function, we don’t need to worry as much about how people will be able to use it. The language brings facilities to import and call this function from other functions, and so forth. Sure, sometimes the quality of those features varies: languages that allow functions to be treated as values and wrapped up as closures versus those that treat them more primitively, for example. But as a function-writer, all that is beyond our scope of concern.

But we don’t always have that luxury when programming. This most often appears when dealing with configuration files formats, or small domain-specific languages. Often, these start small and straight-forward, but users then use them for ever more complicated tasks, until the lack of abstraction becomes a huge problem.

The old classic example of this was configuring mail or HTTP servers. Consider Sendmail, which has an awful reputation for good reason. Its “configuration” ballooned into a miniature programming language compiled via m4 macros. It’s the classic example of the degenerate case: you start with a configuration format, and eventually it turns into a shitty interpreter of a shitty programming language. Oops.

No wonder some cynics believe everything ends up with “an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.” (Greenspun’s tenth rule)

The problem arises because of a simple progression:

  1. We need a small configuration file. Easily done!
  2. Our configurations are becoming very large, we need some abstraction capabilities. Some ad-hoc ones can be added!
  3. Eventually, we need better ones than that for complicated situation XYZ. Well, have some more ad-hoc features?
  4. …
  5. A mess.

This progression is something of an anti-design approach. When applied to programming languages, we end up with something like PHP or Javascript. The core problem here is that we generally can’t break backward compatibility, so the process affords us no way to refactor our designs. The design can never actually be improved upon, only added to. A naive design got solidified too soon, and the problems are only discovered after they cannot be fixed anymore.

For configuration files, the core problem is almost always the lack of anticipation of the need for abstraction. A flat configuration undeniably works, it’s just that in the real world, many configurations begin to contain a lot of repetition and grow to unmanageable sizes. Moreover, larger configurations tend to require us to split responsibilities: multiple teams might need to manage different parts of it, or even obtain parts of the configuration from third-parties.

So our design constraints are something like this:

  • A flat configuration can do the job.
  • But we have multiple teams needing to manage different parts.
  • We need to obtain some parts from third-parties.
  • There are a lot of repeated chunks of configuration we could use templates for.
  • There are other “shapes” of duplication besides just basic repetition. Tree-like shapes, for example.
  • We might like to integrate with other tools. For example, to dynamically discover configuration data.
  • More and more special cases are likely to be discovered in the future.
  • We still want to be able to analyze the configuration. To see what’s changing from state to state, for example.

I know of three good ways to manage this situation:

  1. Leave configurations flat. Instead encourage development of external configuration-generating tools.
  2. Develop an embedded DSL. Sacrifice the analysis advantages of a declarative format, and instead just embrace the fact that configuration is complicated, and use a real programming language that was actually reasonably well-designed.
  3. Use a declarative intermediate. Run arbitrary code that generates a declarative intermediate flat configuration, then use that.

In a way, all three get at the same thing. With the first approach, any tool can generate the flat configuration. With the third approach, that tool gets built in as part of the program in question. With the second approach, we don’t necessarily have a configuration available as a data type, but the state that the code gets put in by executing the configuration DSL could in principle be serialized.

Half-measures

Sometimes, only some of these problem actually need addressing. Many services in Linux distributions now use a simple approach to handling the team-separation and third-party issues. Instead of using a single file, the tool uses a whole directory for storing configuration information. (Usually tool.d instead of tool.conf.) Separate teams can manage separate files, and third-parties (often the distribution itself) can drop their own files in the directory.

We see something similar with Apache’s approach to allowing distributed configuration. An .htaccess file spreads some configuration responsibilities out into different users and groups with different permissions.

Let’s take a look at a few examples of tools grappling with complex configurations.

Terraform

Terraform is a tool for describing and standing up cloud services. It’s pretty much a small declarative programming language for infrastructure. (Part of the “infrastructure as code” modern trend.) Someday, perhaps I should write something up about it’s design. It’s really quite interesting.

This tool has to solve the configuration problem, too. It fits pretty directly into the mold: a flat configuration would obviously work, but would become unmanageable for larger projects.

First, Terraform offers a small ad-hoc abstraction: you can repeat a resource several times by specifying a count. This helps quite a lot, since the first source of duplication anyone is likely to encounter is a result of needing to start multiple servers running the same service. This kind of abstraction is pretty much inevitable going from the simplest setup to anything even slightly more sophisticated.

After that, Terraform offers just one additional abstraction feature: modules. Modules serve several purposes at once:

  1. It lets configurations be subdivided across multiple teams.
  2. It’s a simple way of obtaining bits of configuration from third-parties.
  3. It acts as an extra abstraction mechanism. You can think of modules as templates or extremely primitive functions.

At this point, the internal configuration language of Terraform stops. If this level of abstraction isn’t good enough, you have two options (still involving Terraform):

  1. You can generate Terraform configurations from another language. Either ad-hoc, or perhaps going all the way to creating a Ruby embedded DSL for generating Terraform configurations.
  2. For some kinds of problems, perhaps you can simply write a new plug-in for Terraform.

Terraform doesn’t seem to be doing too badly here. Part of that I think is forced by its design: it really requires a fully declarative intermediate, as part of its “planning” process for figuring out what changes to execute. So I doubt it will ever lose access to a nice declarative configuration.

Jenkins 1.x

Since the 1.x series of Jenkins has now been replaced aggressively by the 2.x series attempting to fix many of its design flaws, I think I can safely trash this garbage fire without upsetting anyone. This is clearly a good example of a design that just accumulated into a terrible monster.

Jenkins is a project clearly designed to be extensible. Aggressively so. Even core parts of the project were essentially just plug-ins. If you’ve read my earlier essay on what makes good design, you might raise an eyebrow at this. It was a disaster.

Jenkins job configuration was essentially just a flat list of projects. The only ability to create abstractions was through developing new Jenkins plug-ins. Since this was rather difficult, few actually did it. Plug-ins were only created for a few archetypes of projects.

Instead, people used the generic “run a script” project type, and plug-ins accumulated as ad-hoc tools to try to make Jenkins do things that the bash scripts couldn’t do themselves, since they were confined to a workspace.

Since this meant that there were no abstraction mechanisms (outside creating plug-ins) available to users at all, the best approach to configuring Jenkins 1.x was to write your own ad-hoc job XML generation scripts. Possibly because the plug-in situation was such a mess (constant, unending breaking changes), there never seemed to be any truly good tools for this purpose. I, at least, always found it more convenient to just jumble some strings together into XML with a Python script. It was pretty dismal.

Jenkins 2.x

The 2.x series set out to fix all these problems with a pretty radical redesign:

  1. “Multi-branch pipelines” allow Jenkins to auto-generate new jobs for each project. This not only allows “distributed configuration” by getting job configuration from Jenkinsfile in each repository, instead of solely from Jenkins itself, it also automatically integrates configuration data from the most important external tool: your repositories (in GitHub, or wherever else).
  2. It switches over to a completely new model of job configuration. Now builds can be scripted with a Groovy-based embedded DSL (with code in that Jenkinsfile). This pretty much embraced the idea that builds needed a real programming language’s worth of control. It also simplified the plug-in model: now plug-ins generally just provide new built-in functions that can be used from Jenkinsfile scripts, instead of functioning in ad-hoc easily broken ways.
  3. Real abstraction capabilities became possible through the traditional language route. Most especially: shared libraries. Jenkins included a library command in its DSL that imports declarations (written in the Groovy DSL) from another repository.

The interesting thing about these shared libraries is that they allowed some of the benefits of declarative jobs descriptions to come back. This is despite job configuration now consisting largely of imperative Groovy scripts.

Jenkins has a “declarative pipeline syntax” that I thought was rather odd, as it didn’t really seem to have any advantages. But a commenter points out to me that there are some UI advantages, however modest. The declarative syntax can handle conditional stages, and can let Jenkins display the full pipeline even if the build failed in a earlier stage.

Each job can forgo directly writing imperative code, and instead call into the shared library with a blob of plain data acting as declarative configuration for that particular job. The shared library has some imperative code, sure, but that’s a single place to manage the abstractions for your organization. Meanwhile, each repository’s Jenkinsfile can just be a small blob of declarative data.

But the design still suffers from some lack of declarativeness. Many features you might like from a continuous integration system (like having a dependency graph of upstream and downstream jobs) have gone away.

What happens in normal languages?

When we encounter a lack of abstractive capabilities in a regular programming language, the outcome is usually similar: we often resort to some kind of code generation. This might be actually generating a source file, or it could be internally via a macro system. Or fancier tools, like Template Haskell, or Java class loaders that generate classes on the fly.

But there’s also another interesting phenomenon: in many cases we can also resort to reflection. This is not just possible for dynamic languages, but static ones like Java, too. This might be interesting to look at in the future.