Doctest parallel: Test interactive Haskell examples

doctest-parallel is a library that checks examples in Haddock comments. It is similar to the popular Python module with the same name.

Installation

doctest-parallel is available from Hackage. It cannot be used as a standalone binary, rather, it expects to be integrated in a Cabal/Stack project. See examples/ for more information on how to integrate doctest-parallel into your project.

Migrating from doctest

See issue #11 for more information.

Usage

Below is a small Haskell module. The module contains a Haddock comment with some examples of interaction. The examples demonstrate how the module is supposed to be used.

module Fib where

-- | Compute Fibonacci numbers
--
-- Examples:
--
-- >>> fib 10
-- 55
--
-- >>> fib 5
-- 5
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)

A comment line starting with >>> denotes an expression. All comment lines following an expression denote the result of that expression. Result is defined by what a REPL (e.g. ghci) prints to stdout and stderr when evaluating that expression.

doctest-parallel will fail on comments that haddock also doesn’t like. Sometimes (e.g., #251), this means that doctest-parallel will fail on input that GHC accepts.

Command line

Usage:
  doctest [ options ]... [<module>]...
  doctest --help
  doctest --version
  doctest --info

Options:
   -jN                      number of threads to use
   --log-level=LEVEL        one of: debug, verbose, info, warning, error. Default: info.
   --ghc-arg=ARG            pass argument to GHC when parsing, pass multiple times for multiple flags
†  --implicit-module-import import module before testing it (default)
†  --randomize-order        randomize order in which tests are run
†  --seed=N                 use a specific seed to randomize test order
†  --preserve-it            preserve the `it` variable between examples
   --nix                    account for Nix build environments (default)
   --quiet                  set log level to `Error`, shorthand for `--log-level=error`
   --verbose                set log level to `Verbose`, shorthand for `--log-level=verbose`
   --debug                  set log level to `Debug`, shorthand for `--log-level=debug`
   --help                   display this help and exit
   --version                output version information and exit
   --info                   output machine-readable version information and exit

Supported inverted options:
   --no-nix
†  --no-implicit-module-import
†  --no-randomize-order (default)
†  --no-preserve-it (default)

Options marked with a dagger (†) can also be used to set module level options, using
an ANN pragma like this:

  {-# ANN module "doctest-parallel: --no-randomize-order" #-}

Example groups

Examples from a single Haddock comment are grouped together and share the same scope. E.g. the following works:

-- |
-- >>> let x = 23
-- >>> x + 42
-- 65

If an example fails, subsequent examples from the same group are skipped. E.g. for

-- |
-- >>> let x = 23
-- >>> let n = x + y
-- >>> print n

print n is not tried, because let n = x + y fails (y is not in scope!).

Setup code

You can put setup code in a named chunk with the name $setup. The setup code is run before each example group. If the setup code produces any errors/failures, all tests from that module are skipped.

Here is an example:

module Foo where

import Bar.Baz

-- $setup
-- >>> let x = 23 :: Int

-- |
-- >>> foo + x
-- 65
foo :: Int
foo = 42

Note that you should not place setup code in between the module header (module ... where) and import declarations. GHC will not be able to parse it (issue #167). It is best to place setup code right after import declarations, but due to its declarative nature you can place it anywhere in between top level declarations as well.

Multi-line input

GHCi supports commands which span multiple lines, and the same syntax works for doctest:

-- |
-- >>> :{
--  let
--    x = 1
--    y = 2
--  in x + y + multiline
-- :}
-- 6
multiline = 3

Note that >>> can be left off for the lines following the first: this is so that haddock does not strip leading whitespace. The expected output has whitespace stripped relative to the :}.

Some peculiarities on the ghci side mean that whitespace at the very start is lost. This breaks the example broken, since the x and y aren’t aligned from ghci’s perspective. A workaround is to avoid leading space, or add a newline such that the indentation does not matter:

{- | >>> :{
let x = 1
    y = 2
  in x + y + works
:}
6
-}
works = 3

{- | >>> :{
 let x = 1
     y = 2
  in x + y + broken
:}
3
-}
broken = 3

Multi-line output

If there are no blank lines in the output, multiple lines are handled automatically.

-- | >>> putStr "Hello\nWorld!"
-- Hello
-- World!

If however the output contains blank lines, they must be noted explicitly with <BLANKLINE>. For example,

import Data.List ( intercalate )

-- | Double-space a paragraph.
--
--   Examples:
--
--   >>> let s1 = "\"Every one of whom?\""
--   >>> let s2 = "\"Every one of whom do you think?\""
--   >>> let s3 = "\"I haven't any idea.\""
--   >>> let paragraph = unlines [s1,s2,s3]
--   >>> putStrLn $ doubleSpace paragraph
--   "Every one of whom?"
--   <BLANKLINE>
--   "Every one of whom do you think?"
--   <BLANKLINE>
--   "I haven't any idea."
--
doubleSpace :: String -> String
doubleSpace = (intercalate "\n\n") . lines

Matching arbitrary output

Any lines containing only three dots (...) will match one or more lines with arbitrary content. For instance,

-- |
-- >>> putStrLn "foo\nbar\nbaz"
-- foo
-- ...
-- baz

If a line contains three dots and additional content, the three dots will match anything within that line:

-- |
-- >>> putStrLn "foo bar baz"
-- foo ... baz

QuickCheck properties

Haddock (since version 2.13.0) has markup support for properties. Doctest can verify properties with QuickCheck. A simple property looks like this:

-- |
-- prop> \n -> abs n == abs (abs (n :: Int))

The lambda abstraction is optional and can be omitted:

-- |
-- prop> abs n == abs (abs (n :: Int))

A complete example that uses setup code is below:

module Fib where

-- $setup
-- >>> import Control.Applicative
-- >>> import Test.QuickCheck
-- >>> newtype Small = Small Int deriving Show
-- >>> instance Arbitrary Small where arbitrary = Small . (`mod` 10) <$> arbitrary

-- | Compute Fibonacci numbers
--
-- The following property holds:
--
-- prop> \(Small n) -> fib n == fib (n + 2) - fib (n + 1)
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)

If you see an error like the following, ensure that QuickCheck is a dependency of your test-suite.

<interactive>:39:3:
    Not in scope: ‘polyQuickCheck’
    In the splice: $(polyQuickCheck (mkName "doctest_prop"))

<interactive>:39:3:
    GHC stage restriction:
      ‘polyQuickCheck’ is used in a top-level splice or annotation,
      and must be imported, not defined locally
    In the expression: polyQuickCheck (mkName "doctest_prop")
    In the splice: $(polyQuickCheck (mkName "doctest_prop"))

Hiding examples from Haddock

You can put examples into named chunks, and not refer to them in the export list. That way they will not be part of the generated Haddock documentation, but Doctest will still find them.

-- $
-- >>> 1 + 1
-- 2

Using GHC extensions

You can enable GHC extensions using the following syntax:

-- >>> :set -XTupleSections

If you want to omit the information which language extensions are enabled from the Doctest examples you can use the method described in Hiding examples from Haddock, e.g.:

-- $
-- >>> :set -XTupleSections

Using GHC plugins

You can enable GHC plugins using the following syntax:

-- >>> :set -fplugin The.Plugin

Hiding Prelude

You hide the import of Prelude by using:

-- >>> :m -Prelude

Per module options

You can override command line flags per module by using a module annotation. For example, if you know a specific module does not support test order randomization, you can disable it with:

{-# ANN module "doctest-parallel: --no-randomize-order" #-}

Test non-exposed modules

Generally, doctest-parallel cannot test binders that are part of non-exposed modules, unless they are re-exported from exposed modules. By default doctest-parallel will fail to do so (and report an error message), because it doesn’t track whether functions are re-exported in such a way. To test a re-exported function, add the following to the non-exposed module:

{-# ANN module "doctest-parallel: --no-implicit-module-import" #-}

This makes doctest-parallel omit the usual module import at the start of a test.

Then, before a test -or in $setup- add:

>>> import Exposed.Module (someFunction)

Relation to doctest

This is a fork of sol/doctest that allows running tests in parallel and aims to provide a more robust project integration method. It is not backwards compatible and expects to be setup differently. At the time of writing it has a few advantages over the base project:

• It runs tests in parallel
• It runs tests against compiled code, instead of reinterpreting your whole project
• It isolates examples in modules, ensuring your tests don’t accidentally rely on each other
• It parses cabal files to discover modules, no need for custom setup anymore!
• A minor change: it does not count lines in setup blocks as test cases
• A minor change: the testsuite has been ported to v2 commands

All in all, you can expect doctest-parallel to run about 1 or 2 orders of magnitude faster than doctest for large projects.

Relation to cabal-docspec

There is no direct relation between doctest-parallel and cabal-docspec. They are similar in some ways:

• Both projects load code from precompiled modules
• Both project aim to get rid of the need for custom setups

And different in others:

• As a fork of doctest, doctest-parallel inherits the testsuite doctest accumulated over the years.
• doctest-parallel parses Cabal project files, instead of parsing files from dist-newstyle. This makes it compatible with Stack, provided a .cabal is still present.
• doctest-parallel uses the GHC API to parse comments. This should in theory be more reliable (though I doubt it will ever matter in practice).
• doctest-parallel runs tests in parallel.

Development

To run the tests:

cabal run spectests
cabal run doctests

Future of this project

• It would be lovely if we could get rid of the needs for write-ghc-environment-files: always option for Cabal. To properly do this, I think Cabal should do two things:
  1. Deprecate GHC environment files as a way to implicitly setup environments. Instead, environment files should be written to the dist-newstyle directory and activated using some subcommand, e.g. cabal shell. This avoids the many problems GHC environment files have, while retaining their functionality for people who like them.
  2. Any subcommands should be run with GHC_ENVIRONMENT set - pointing to the GHC environment file. Like Stack, this would create a hassle free way of using Cabal in combination with projects/executables that use the GHC API (e.g., clash-ghc, doctest-parallel).
• It would be nice if Cabal would expose more information by default (probably through auto-generated modules) in order for doctest-parallel to properly work. Specifically, it needs to know the exact default-extensions, ghc-options, and CPP flags the project is compiled with. These options are obtainable by using a custom Setup.hs, but this has its own list of problems.
  • Alternatively, if comments could be included in and loaded from .hi files that’d solve all issues too.
• Hopefully many of the improvements made here can make their way back into sol/doctest.

Of course, if you wish to add a feature that’s not in this list, please feel free top open a pull request!

Contributors

• Adam Vogt
• Anders Persson
• Ankit Ahuja
• Edward Kmett
• Hiroki Hattori
• Joachim Breitner
• João Cristóvão
• Julian Arni
• Kazu Yamamoto
• Levent Erkok
• Luke Murphy
• Matvey Aksenov
• Michael Orlitzky
• Michael Snoyman
• Nick Smallbone
• Phil de Joux
• Sakari Jokinen
• Simon Hengel
• Sönke Hahn