Caveat Configurator: When Configuration Becomes Code

At Jane Street, we configure systems under two constraints: flexibility and safety. We trade many instruments around the world using mostly in-house tools, so each engineer is responsible for many systems. Reusing a general system with small changes can make the difference between launching an opportunity quickly and building a new system from scratch.

We also trade our own money. A small bug may cost an opportunity or prevent us trading somewhere; a large bug in a tight loop can threaten the firm itself. Configuration is one way to get more mileage from general systems while managing those risks.

In 2014, a colleague and I presented ocaml_plugin, a library that makes dynamic loading of OCaml code easier. It let us plug code into a system and change its behavior more flexibly than earlier configuration architectures. We were excited enough to use it in a new implementation of our email system.

Our first, naive use of plugins in production turned out to be a disaster. I want to explain the problems we encountered, how we arrived at plugins, and the approaches we eventually used to deal with configuration.

Configuration lets you change a system's behavior without necessarily recompiling, repackaging and redeploying the application. It separates decisions exposed to operators from core functionality, and it can provide an escape hatch for situations the original authors did not anticipate.

Consider a mail server: it accepts connections, receives messages, processes them—for example by adding headers or removing infected attachments—and delivers or forwards them. One possible design has no separate configuration at all: changes are made in the application, which is rebuilt and deployed.

Zero configuration is not crazy. If sensible defaults and easy deployment meet your needs, it avoids many of the problems discussed here. Mail systems often need changes at runtime, but it is worth asking whether separate configuration is needed before introducing it.

As we move more decisions out of the application's core and into configuration, the system can become more flexible. At the extreme, configuration handles almost everything. Whatever the boundary, the choices must be represented somewhere the application can read, such as a file or database.

A common starting point is pure data: command-line parameters, environment variables, key-value files, or formats such as YAML, JSON and S-expressions. A small flat file is easy to understand: if it says the mail server listens on port 25, there is no hidden logic to interpret.

Simple data is also relatively easy to version. An upgrader can read an old format and produce the internal representation needed by a newer application. The drawback is limited expressive power: as configuration grows more complex, it can become verbose and repetitive.

In a JSON-like mail configuration, we might list ports, a log directory and level, then specify next-hop servers by sender domain, including retry intervals. Adding another domain can duplicate ports and retry schedules. We can factor repeated values into defaults, but some fields have several common cases rather than one obvious default.

The tempting next step is to add macros or variables. That can help repetition, but it begins an ad hoc language. You might instead deliberately design a domain-specific language, choosing its structure and features up front rather than accumulating special cases.

A DSL gives its designer control and can be arbitrarily powerful. But language design takes time and is hard even for skilled designers. The resulting parser or interpreter can have bugs that are difficult to distinguish from application bugs, especially during a production incident.

The semantics may be unfamiliar to everyone else, even if they draw on features from existing languages. Tooling such as syntax highlighting, navigation, completion and tests must be created or adapted. Experienced mail operators still have to learn this new language, and its author may be the only person called when it breaks at 2 a.m.

For a concrete example, consider Exim. A basic configuration can deliver mail to a file named for the local part of an address, check a user list and reject unknown users. Adding Gmail-style plus addressing lets a recipient append a suffix to an address while retaining the base destination.

Even this modest feature requires a more involved Exim configuration if we allow individual users to enable it. There are nested braces, a special suffix pattern, and different rules for referring to or checking variables. Ordering and scope matter. The example may work, but getting it right takes real effort.

We already have a tool designed to express complex behavior: a programming language, with libraries and workflows we know. In OCaml, we can model a message with sender, recipients and body, then define a handler that returns delivered, relay with destinations, or reject with an error.

To implement plus addressing, the handler can split the local part around the plus sign, check the user's preference, and either reject or deliver the message. For someone familiar with the language, that is ordinary code rather than another set of DSL-specific rules. Dynamic loading is possible in many languages; ocaml_plugin makes it convenient in OCaml.

Initially, replacing larger configurations with OCaml made them concise and readable. Then people used the power they had been given: plugin usage grew and the problems appeared. Code can do almost anything, and a plugin configuring a production service runs inside its process, under its user and on its machine.

We deployed systems whose users wrote their own plugins separately from the application developers. Things worked until we needed to change and redeploy the core application. We had not thought enough about versioning: a plugin must compile against the libraries linked into the particular application version.

If library interfaces change and the plugin is not updated, it may no longer compile at runtime. We actually abandoned and restarted some applications after encountering this problem. Independent plugin files also need a plan for where they live, how they are deployed and tracked, and how they are updated alongside the application.

Surprisingly, the tools were still poor. A standalone plugin outside the normal codebase does not automatically tell an editor where its dependencies live, so features such as jump-to-definition may fail. Treating configuration code differently from the rest of the code breaks assumptions in the tools.

That matters even more when configuration grows from a 12- or 13-line example to 4,000 lines of OCaml, as our actual mail-server configuration did. At that scale, a poor development workflow is no longer a minor inconvenience.

In many cases we moved the boundary back: we bundled the plugins with the application and deployed them together, even though we still loaded them through ocaml_plugin. We retained a clean separation between core and configurable behavior and an emergency runtime escape hatch, but lost much of the original independent configuration benefit.

We took two broad routes. For some applications, we concluded independent plugin flexibility was unnecessary and shipped plugins with the application. Elsewhere, we invested in tools for editing, version control, rebasing and merging dispersed plugin changes, effectively rebuilding parts of an ordinary development environment.

Code has real strengths: it is expressive, scales through familiar abstractions, and has mature tools. It also comes with a culture of review, tests and refactoring. Those practices improve quality and correctness, yet people do not automatically apply them to configuration files.

One alternative is configuration generation. Write code to produce a finite, static configuration, then have the application read that data. This combines the expressive power of code when authoring the configuration with the transparency of inspecting the result before it is loaded.

Generation requires a configuration that can be written down finitely; a live function cannot simply be serialized as ordinary data. The on-disk format still needs a versioning story, but upgrading a data format is simpler than maintaining compatibility between dynamically loaded code and all of an application's dependencies.

A system already using hand-written data can move to a generator without necessarily changing the application: the generator emits the same format. The technique also works with third-party systems, since the generator can use any language and output the format the target application expects.

At Jane Street, one option is an OCaml script pinned to appropriate compiler and library versions. Another is a repository with code review, a build daemon and tests. Committing the generated configuration as well makes changes easy to inspect and allows rollback without running the generator again.

Our monitoring configuration describes hosts, services, check commands and schedules. Because the generated file is visible before loading, operators can inspect the actual settings. We can also generate a CSV of resolved hosts and checks, then diff versions to see exactly which checks disappear or which hosts and checks are added.

The other route is to tackle plugin problems directly. We need to handle version compatibility, the danger of a buggy plugin or top-level exception destabilizing the application, inconsistent deployment and file management, and poor tooling. Jane Street built a system called plugd to address these issues.

Plugd centralizes configurations in a repository and treats them like application code. It serves versions that have been reviewed, compiled and tested, while decoupling the application from the location of plugin files. It also provides a workflow for moving patches between versions so plugins used by older application releases can remain up to date.

That infrastructure took real work, but plugins provide something configuration generation cannot: runtime dynamism. A function can be the configuration, so behavior need not be limited to a finite static file.

The decision tree begins by asking whether runtime computational flexibility is necessary. If not, configuration generation is a strong choice. If it is, ask whether plugin writers must be decoupled from application developers; if not, bundle and deploy the plugins with the application.

If independent plugins are necessary and stability matters, use disciplined versioning and infrastructure rather than naive plugins. Also reconsider whether plugins are needed at all: in many cases their attractive separation of core and configurable functionality could have been achieved simply by structuring the application well.

Asked how plugd itself is configured, I would put it closer to the zero-configuration camp. It has sensible defaults and integrates tightly with the code-review system and build daemon, from which it can obtain much of the information it needs.

An audience member asks about researchers and operations staff outside the developer group. Plugins require their authors to know OCaml. Although many non-engineers at Jane Street can read or write code, we sometimes deliberately keep a simpler format that a person can open and change without learning a programming language.

The two approaches can also be combined: a plugin provides runtime dynamism while defining a simpler configuration format inside it. In other cases traders write plugins themselves, and people familiar with the core application review them to help ensure they are safe and reasonable.

A question about versioning asks whether the issue is merely a lack of version control. Version control matters, and configuration is less likely to be placed under it by default, but the harder issue is compatibility between a plugin and the precise library versions available to an application at runtime.

OCaml plugin bundles the compiler and dependencies into the application. At runtime it expands them, compiles the plugin against those dependencies and dynamically loads the result. A plugin written against a different version of a library may therefore fail to compile when loaded by that application.

Another alternative is to expose the application as a library and let users build a small program around it. We use related patterns extensively: our codebase is a collection of libraries, and some expose functions for starting configured applications. The boundary between configuration and using a library is not always sharp.

When should you choose that approach? The answer depends partly on how much functionality is bundled and how much is left to the user. An audience member adds that if behavior must change while a program is executing, loading a new plugin matters; if everything is set up before running, the difference may mostly be tooling.

Asked about sandboxing, I describe libraries that start worker processes to build plugins so a failing worker does not directly affect the master. This adds substantial plumbing and has not become our dominant use case, though it remains an option.

A 'good' plugin is one that passes the workflow: code review, compilation and tests. The build daemon runs the relevant tests for each release, and plugd builds and tests plugins independently before serving them to the application.

The audience asks about the people needed to build and maintain plugd. The speakers estimate that roughly one person built most of it, with maintenance requiring a fraction of a person's time, although the total development effort across contributors was considerable.

What happens if the owner is unavailable? That is a broader operational problem. Other people understand the production operation well enough to respond, and code review spreads some understanding of changes. Documentation is encouraged but uneven; for especially important systems, we make sure several people keep up with day-to-day operation.

Finally, someone asks whether plugin functionality ever migrates into the core application. I expect that it does, though I cannot immediately name a specific example.