wsk: A Straightforward and Maintainable Build System from the Bloomberg Graphics Team

An introduction to wsk, a straightforward and maintainable build system, by one of its contributors, data journalist Michael Keller of the Bloomberg Graphics team

At the start of 2016, we on the Bloomberg News Graphics team were looking to build a new project creation system. Our setup at the time largely left it up to the project creator to manage how files were built. While this gave creators flexibility, we thought a little more structure and common ground would be helpful.

Our goals

Flexibility was an important element to preserve because our team is composed of journalists, designers and developers who create visual news stories on deadline. Each project might be different and need modifications to react to an unfolding news event.

Our build system would have to meet the following requirements (in no particular order):

  • Be flexible and extensible for people on our team unfamiliar with the latest build system frameworks.
  • Provide rich feedback so that if the user did something wrong, it would try to help them correct it.
  • Make clear what was a problem with the project versus a problem with the build system. This was critical during the adoption phase.
  • Not close off possibilities or lock us into one way of doing things.
  • Each of our projects is a custom design. We didn’t want a system that constrained creativity or had an architecture that slowed down implementation of new features (such as waiting for dependencies to exist or be updated).

What was out there

The most common choices, at the time, were Gulp and Webpack. We ended up not choosing either of them because each had elements that went against at least one of our goals.

  • Writing a Gulp plugin, for example, requires knowledge of streams, which is a barrier to entry.
  • Console feedback tells you that a task happened and how long it took but not what it did, specifically.
  • Gulp requires an ecosystem of plugins. If you want to use Sass, you rely on gulp-sass, which might not be up to date or it might introduce its own bugs.
  • Webpack tends to take full ownership over the whole process; breaking off pieces of it or adding to it can be difficult. This might close off possibilities or require us to find “the Webpack way” of doing a given transformation if a project creator wanted it done.

Where we started

Our first approach in service of these goals was to use vanilla npm scripts and the command-line interfaces for the libraries we wanted to use. We used chokidar-cli to trigger actions when files changed. Our package.json looked like this:

The functionality was okay, but we saw very inconsistent console output.

This concern goes beyond aesthetics. If the console is sending differently-styled output on changes, the user has to interpret multiple signals. This is distracting and makes it harder to recognize actionable feedback.

It would be better if normal compile messages were in a consistent format. Errors should appear in a different color and with a stack trace, if possible, to break up the formatting. The user can detect these changes more easily and attend to them.

Cognitively, the eye is better at noticing movement in the periphery of vision than it is at catching detail. If the build console is in the corner of the screen monitor, it’s easier to notice a disruption if there’s a break in consistency than if you have a number of variable-length messages printing all the time.

Since we were implementing a build system where one previously did not exist, clear notifications were important to separate system errors from project errors. It’s very easy to blame the new, unknown thing when there’s an error. Descriptive build notifications can help the user diagnose a problem with the project more quickly.

Improving console outputs

The first attempt at trying to make console outputs more consistent was piping them to a script that would standardize them. Here’s the new output:

The problems:

  1. This was a lot of work that would likely need to be redone if a library modified its output style.
  2. It would require more work if we changed libraries, in opposition to our first goal.
  3. It made our package.json very difficult to read, also in opposition to that goal for anyone who wanted to edit their npm scripts. Here’s what that looked like:

Because each of these scripts handles its output differently, we had to pipe console messages all over the place. Some went to /dev/null; others used 2>&1 to redirect stderr to stdout. It was a mess.

A better setup

These experiments led us to create wsk and wsk-notify, which is our current setup. wsk is a watcher specification around chokidar that makes it easier to declare a glob of files to watch and vanilla JavaScript modules to run when those files change. wsk-notify is a module for standardized console and desktop notifications.

This setup works with our goals because:

  1. No special knowledge is required to write a task file (a “plugin” using Gulp vocabulary).
  2. By using chokidar to watch files, we get access to a diverse set of events. Our notifications can report exactly what events are happening and what tasks are being dispatched as a result.
  3. By using libraries directly, we avoid intermediate plugins as much as we can.

Although we passed over using command-line APIs for other reasons, we found that using libraries directly via their JavaScript API also increased the number of options we had when using them. That makes intuitive sense, as you can only squeeze so much functionality into command-line flags.

Usage

An example wsk watcher file looks something like this:

// sass-watch.js

var watcher = require('wsk').watcher;

var watchGroup = {
  serviceName: 'sass',
  path: 'src/css/*.scss',
  events: [
    {
      type: 'change',
      taskFiles: 'build/tasks/sass/onChange.js',
      options: {
        foo: true
      }
    },
    {
      type: 'change',
      taskFiles: 'build/tasks/sass/onAdd.js'
    },
    {
      type: 'unlink',
      taskFiles: 'build/tasks/sass/onUnlink.js'
    }
  ]
};

watcher.add(watchGroup);

The only requirement for a task file is that it export an onEvent function which takes the event type listed in type above, the path of the file that changed and any options specified. Anything you put in that function is up to you.

// onChange.js
var notify = require('wsk').notify;

module.exports = function (eventType, filePath, options) {
  // Put library transform here
  // Write the transformed file with `fs.writeFile`, for example
  // Notify user the file was written
  notify({
    message: 'Compiled CSS file...'  ,
    value: filePath.replace('src', 'public'),
    display: 'compile'
  });
};

Conclusion

We built wsk to address our mix of skills and constraints. It might not be the right solution for everyone, but if you’ve run into issues with other plugins, been constrained by adding extra concepts into your pipeline (such as streams), or been frustrated when you want to use a new library and had to look for a “How to use with Webpack/Gulp/Grunt” section of the readme, wsk might be helpful for you.

Over the last 18 months of usage, it’s proven to be a flexible and maintainable system. It has powered projects as simple as one-off graphics to multi-day or multi-page series that function more like static-site generators.

You can find usage examples and more documentation on the wsk and wsk-notify sites.