SRFI-64 Conditonal Test Suites and Other Advanced Features

The following describes features for controlling which tests to execute, or specifying that some tests are expected to fail.

Test specifiers

Sometimes we want to only run certain tests, or we know that certain tests are expected to fail. A test specifier is a one-argument function that takes a test-runner and returns a boolean. The specifier may be run before a test is performed, and the result may control whether the test is executed. For convenience, a specifier may also be a non-procedure value, which is coerced to a specifier procedure, as described below for count and name.

A simple example is:

(if (var some-condition)  (test-skip 2)) ;; skip next 2 tests

Scheme Procedure: test-match-name name

The resulting specifier matches if the current test name (as returned by test-runner-test-name) is equal? to name.

Scheme Syntax: test-match-nth n [count]

This evaluates to a stateful predicate: A counter keeps track of how many times it has been called. The predicate matches the n’th time it is called (where 1 is the first time), and the next ‘(- count 1)’ times, where count defaults to 1.

Scheme Syntax: test-match-any specifier …

The resulting specifier matches if any specifier matches. Each specifier is applied, in order, so side-effects from a later specifier happen even if an earlier specifier is true.

Scheme Syntax: test-match-all specifier …

The resulting specifier matches if each specifier matches. Each specifier is applied, in order, so side-effects from a later specifier happen even if an earlier specifier is false.

count (i.e. an integer) Convenience short-hand for: ‘(test-match-nth 1 count)’.

name (i.e. a string) Convenience short-hand for ‘(test-match-name name)’.

Skipping selected tests

In some cases you may want to skip a test.

Scheme Syntax: test-skip specifier

Evaluating test-skip adds the resulting specifier to the set of currently active skip-specifiers. Before each test (or test-group) the set of active skip-specifiers are applied to the active test-runner. If any specifier matches, then the test is skipped.

For convenience, if the specifier is a string that is syntactic sugar for (test-match-name specifier). For example:

(test-skip "test-b")
(test-assert "test-a")   ;; executed
(test-assert "test-b")   ;; skipped

Any skip specifiers introduced by a test-skip are removed by a following non-nested test-end.

(test-begin "group1")
(test-skip "test-a")
(test-assert "test-a")   ;; skipped
(test-end "group1")      ;; Undoes the prior test-skip
(test-assert "test-a")   ;; executed

Expected failures

Sometimes you know a test case will fail, but you don’t have time to or can’t fix it. Maybe a certain feature only works on certain platforms. However, you want the test-case to be there to remind you to fix it. You want to note that such tests are expected to fail.

Scheme Syntax: test-expect-fail specifier

Matching tests (where matching is defined as in test-skip) are expected to fail. This only affects test reporting, not test execution. For example:

(test-expect-fail 2)
(test-eqv ...) ;; expected to fail
(test-eqv ...) ;; expected to fail
(test-eqv ...) ;; expected to pass