SRFI-64 Writing a New Test Runner ​

This section specifies how to write a test-runner. It can be ignored if you just want to write test-cases.

Call-back Functions ​

These call-back functions are “methods” (in the object-oriented sense) of a test-runner. A method test-runner-on-event is called by the implementation when event happens.

To define (set) the callback function for event use the following expression. (This is normally done when initializing a test-runner.)

(test-runner-on-event! runner event-function)

An event-function takes a test-runner argument, and possibly other arguments, depending on the event.

To extract (get) the callback function for event do this: (test-runner-on-event runner)

To extract call the callback function for event use the following expression. (This is normally done by the implementation core.) ‘((test-runner-on-event runner) runner other-args …)’.

The following call-back hooks are available.

Scheme Procedure: test-runner-on-test-begin runner ​

Scheme Procedure: test-runner-on-test-begin! runner on-test-begin-function ​

Scheme Procedure: on-test-begin-function runner ​

The on-test-begin-function is called at the start of an individual testcase, before the test expression (and expected value) are evaluated.

Scheme Procedure: test-runner-on-test-end runner ​

Scheme Procedure: test-runner-on-test-end! runner on-test-end-function ​

Scheme Procedure: on-test-end-function runner ​

The on-test-end-function is called at the end of an individual testcase, when the result of the test is available.

Scheme Procedure: test-runner-on-group-begin runner ​

Scheme Procedure: test-runner-on-group-begin! runner on-group-begin-function ​

Scheme Procedure: on-group-begin-function runner suite-name count ​

The on-group-begin-function is called by a test-begin, including at the start of a test-group. The suite-name is a Scheme string, and count is an integer or #f.

Scheme Procedure: test-runner-on-group-end runner ​

Scheme Procedure: test-runner-on-group-end! runner on-group-end-function ​

Scheme Procedure: on-group-end-function runner ​

The on-group-end-function is called by a test-end, including at the end of a test-group.

Scheme Procedure: test-runner-on-bad-count runner ​

Scheme Procedure: test-runner-on-bad-count! runner on-bad-count-function ​

Scheme Procedure: on-bad-count-function runner actual-count expected-count ​

Called from test-end (before the on-group-end-function is called) if an expected-count was specified by the matching test-begin and the expected-count does not match the actual-count of tests actually executed or skipped.

Scheme Procedure: test-runner-on-bad-end-name runner ​

Scheme Procedure: test-runner-on-bad-end-name! runner on-bad-end-name-function ​

Scheme Procedure: on-bad-end-name-function runner begin-name end-name ​

Called from test-end (before the on-group-end-function is called) if a suite-name was specified, and it did not that the name in the matching test-begin.

Scheme Procedure: test-runner-on-final runner ​

Scheme Procedure: test-runner-on-final! runner on-final-function ​

Scheme Procedure: on-final-function runner ​

The on-final-function takes one parameter (a test-runner) and typically displays a summary (count) of the tests. The on-final-function is called after called the on-group-end-function correspondiong to the outermost test-end. The default value is test-on-final-simple which writes to the standard output port the number of tests of the various kinds.

The default test-runner returned by test-runner-simple uses the following call-back functions:

Scheme Procedure: test-on-test-begin-simple runner ​

Scheme Procedure: test-on-test-end-simple runner ​

Scheme Procedure: test-on-group-begin-simple runner suite-name count ​

Scheme Procedure: test-on-group-end-simple runner ​

Scheme Procedure: test-on-bad-count-simple runner actual-count expected-count ​

Scheme Procedure: test-on-bad-end-name-simple runner begin-name end-name ​

You can call those if you want to write your own test-runner.

Test-runner components ​

The following functions are for accessing the other components of a test-runner. They would normally only be used to write a new test-runner or a match-predicate.

Scheme Procedure: test-runner-pass-count runner ​

Returns the number of tests that passed, and were expected to pass.

Scheme Procedure: test-runner-fail-count runner ​

Returns the number of tests that failed, but were expected to pass.

Scheme Procedure: test-runner-xpass-count runner ​

Returns the number of tests that passed, but were expected to fail.

Scheme Procedure: test-runner-xfail-count runner ​

Returns the number of tests that failed, and were expected to pass.

Scheme Procedure: test-runner-skip-count runner ​

Returns the number of tests or test groups that were skipped.

Scheme Procedure: test-runner-test-name runner ​

Returns the name of the current test or test group, as a string. During execution of test-begin this is the name of the test group; during the execution of an actual test, this is the name of the test-case. If no name was specified, the name is the empty string.

Scheme Procedure: test-runner-group-path runner ​

A list of names of groups we’re nested in, with the outermost group first.

Scheme Procedure: test-runner-group-stack runner ​

A list of names of groups we’re nested in, with the outermost group last. (This is more efficient than test-runner-group-path, since it doesn’t require any copying.)

Scheme Procedure: test-runner-aux-value runner ​

Scheme Procedure: test-runner-aux-value! runner on-test ​

Get or set the aux-value field of a test-runner. This field is not used by this API or the test-runner-simple test-runner, but may be used by custom test-runners to store extra state.

Scheme Procedure: test-runner-reset runner ​

Resets the state of the runner to its initial state.

Example ​

This is an example of a simple custom test-runner. Loading this program before running a test-suite will install it as the default test runner.

emacs-lisp
(define (my-simple-runner filename)
  (let ((runner (test-runner-null))
	(port (open-output-file filename))
        (num-passed 0)
        (num-failed 0))
    (test-runner-on-test-end! runner
      (lambda (runner)
        (case (test-result-kind runner)
          ((pass xpass) (set! num-passed (+ num-passed 1)))
          ((fail xfail) (set! num-failed (+ num-failed 1)))
          (else #t))))
    (test-runner-on-final! runner
       (lambda (runner)
          (format port "Passing tests: ~d.~%Failing tests: ~d.~%"
                  num-passed num-failed)
	  (close-output-port port)))
    runner))
(test-runner-factory
 (lambda () (my-simple-runner "/tmp/my-test.log")))