Test Generators

A Test Generator is a track-specific piece of software to automatically generate a practice exercise's tests. It does this by converting the exercise's JSON test cases to tests in the track's language.

Benefits

Some benefits of having a Test Generator are:

1. Exercises can be added faster
2. Automates "boring" parts of adding an exercise
3. Easy to sync tests with latest canonical data

Use cases

In general, one runs a Test Generator to either:

1. Generate the tests for a new exercise
2. Update the tests of an existing exercise

Generate tests for new exercise

Adding a Test Generator for a new exercise allows one to generate its tests file(s). Provided the Test Generator itself has already been implemented, generating the tests for the new exercise will be (far) less work than writing them from scratch.

Update tests of existing exercise

Once an exercise has a Test Generator, you can re-run it to update/sync the exercise with its latest canonical data. We recommend doing this periodically, to check if there are problematic test cases that need to be updated or new tests you might want to include.

Starting point

There are two possible starting points when implementing a Test Generator for an exercise:

1. The exercise is new and thus doesn't have any tests
2. The exercise already exists and thus has existing tests

Design

Broadly speaking, test files are generated using either:

• Code: the tests files are (mostly) generated via code
• Templates: the tests files are (mostly) generated using templates

We've found that the code-based approach will lead to fairly complex Test Generator code, whereas the template-based approach is simpler.

What we recommend is the following flow:

1. Read the exercise's canonical data
2. Exclude the test cases that are marked as include = false in the exercise's tests.toml file
3. Convert the exercise's canonical data into a format that can be used in a template
4. Pass the exercise's canonical data to an exercise-specific template

The key benefit of this setup is that each exercise has its own template, which:

• Makes it obvious how the test files are generated
• Makes them easier to debug
• Makes it safe to edit them without risking breaking another exercise

Implementation

The Test Generator is usually (mostly) written in the track's language.

Canonical data

The core data the Test Generator works with is an exercise's canonical-data.json file. This file is defined in the exercism/problem-specifications repo, which defines shared metadata for many Exercism's exercises.

Structure

Canonical data is defined in a JSON object. This object contains a "cases" field which contains the test cases. These test cases (normally) correspond one-to-one to tests in your track.

Each test case has a couple of properties, with the description, property, input value(s) and expected value being the most important ones. Here is a (partial) example of the canonical-data.json file of the leap exercise:

{
  "exercise": "leap",
  "cases": [
    {
      "uuid": "6466b30d-519c-438e-935d-388224ab5223",
      "description": "year not divisible by 4 in common year",
      "property": "leapYear",
      "input": {
        "year": 2015
      },
      "expected": false
    },
    {
      "uuid": "4fe9b84c-8e65-489e-970b-856d60b8b78e",
      "description": "year divisible by 4, not divisible by 100 in leap year",
      "property": "leapYear",
      "input": {
        "year": 1996
      },
      "expected": true
    }
  ]
}

The Test Generator's main responsibility is to transform this JSON data into track-specific tests. Here's how the above JSON could translate into Nim test code:

import unittest
import leap

suite "Leap":
  test "year not divisible by 4 in common year":
    check isLeapYear(2015) == false

  test "year divisible by 4, not divisible by 100 in leap year":
    check isLeapYear(1996) == true

The structure of the canonical-data.json file is well documented and it also has a JSON schema definition.

Nesting

Some exercises use nesting in their canonical data. This means that each element in a cases array can be either:

1. A regular test case (no child test cases)
2. A grouping of test cases (one or more child test cases)

Here is an example of nested test cases:

{
  "cases": [
    {
      "uuid": "e9c93a78-c536-4750-a336-94583d23fafa",
      "description": "data is retained",
      "property": "data",
      "input": {
        "treeData": ["4"]
      },
      "expected": {
        "data": "4",
        "left": null,
        "right": null
      }
    },
    {
      "description": "insert data at proper node",
      "cases": [
        {
          "uuid": "7a95c9e8-69f6-476a-b0c4-4170cb3f7c91",
          "description": "smaller number at left node",
          "property": "data",
          "input": {
            "treeData": ["4", "2"]
          },
          "expected": {
            "data": "4",
            "left": {
              "data": "2",
              "left": null,
              "right": null
            },
            "right": null
          }
        }
      ]
    }
  ]
}

Input and expected values

The contents of the input and expected test case keys vary widely. In most cases, they'll be scalar values (like numbers, booleans or strings) or simple objects. However, occasionally you'll also find more complex values that will likely require a bit of preprocessing, such as lambdas in pseudo code, lists of operations to perform on the students code and more.

Scenarios

Test cases have an optional scenarios field. This field can be used by the test generator to special case certain test cases. The most common use case is to ignore certain types of tests, for example tests with the "unicode" scenario as your track's language might not support Unicode.

The full list of scenarios can be found here.

Reading canonical-data.json files

There are a couple of options to read the canonical-data.json files:

1. Fetch them directly from the problem-specifications repository (e.g. https://raw.githubusercontent.com/exercism/problem-specifications/main/exercises/leap/canonical-data.json).
2. Add the problem-specifications repo as a Git submodule to the track repo.
3. Read them from the configlet cache. The location depends on the user's system, but you can use configlet info -o -v d | head -1 | cut -d " " -f 5 to programmatically get the location.

Track-specific test cases

If your track would like to add some additional, track-specific test cases (which are not found in the canonical data), one option is to creating an additional-test-cases.json file, which the Test Generator can then merge with the canonical-data.json file before passing it to the template for rendering.

Templates

The template engine to use will likely be track-specific. Ideally, you'll want your templates to be as straightforward as possible, so don't worry about code duplication and such.

The templates themselves will get their data from the Test Generator on which they iterate over to render them.

Using configlet

configlet is the primary track maintenance tool and can be used to:

• Create the exercise files for a new exercise: run bin/configlet create --practice-exercise <slug>
• Sync the tests.toml file of an existing exercise: run bin/configlet sync --tests --update --exercise <slug>
• Fetch the exercise's canonical data to disk (this is a side-effect of either of the above commands)

This makes configlet a great tool to use in combination with the Test Generator for some really powerful workflows.

Command-line interface

You'll want to make using the Test Generator both easy and powerful. For that, we recommend creating one or more script files.

Here is an example of a shell script that combines configlet and a Test Generator to quickly scaffold a new exercise:

bin/fetch-configlet
bin/configlet create --practice-exercise <slug>
path/to/test-generator <slug>

Building from scratch

Before you start building a Test Generator, we suggest you look at a couple of existing Test Generators to get a feel for how other tracks have implemented them:

• C#
• Clojure
• Common Lisp
• Crystal
• Emacs Lisp
• F#
• Perl 5
• Pharo Smalltalk
• Python
• Rust
• Swift

If you have any questions, the forum is the best place to ask them. The forum discussions around the Rust and the JavaScript test generators might be helpful too.

Minimum Viable Product

We recommend incrementally building the Test Generator, starting with a Minimal Viable Product. A bare minimum version would read an exercise's canonical-data.json and just pass that data to the template.

Start by focusing on a single exercise, preferrably a simple one like leap. Only when you have that working should you gradually add more exercises.

And try to keep the Test Generator as simple as it can be.

Using or contributing

How to use or contribute to a Test Generator is track-specific. Look for instructions in the track's README.md, CONTRIBUTING.md or the Test Generator code's directory.