doctest-parallel
Test interactive Haskell examples
https://github.com/martijnbastiaan/doctest-parallel#readme
Version on this page: | 0.3.1 |
LTS Haskell 22.33: | 0.3.1.1@rev:1 |
Stackage Nightly 2024-09-07: | 0.3.1.1@rev:1 |
Latest on Hackage: | 0.3.1.1@rev:1 |
doctest-parallel-0.3.1@sha256:f1ec7b7044d156dacf3bc5464c9c692d12f490ec8bb933c6a9007d98c42004d5,5862
Module documentation for 0.3.1
- Test
- Test.DocTest
- Test.DocTest.Helpers
- Test.DocTest.Internal
- Test.DocTest.Internal.Extract
- Test.DocTest.Internal.GhcUtil
- Test.DocTest.Internal.GhciWrapper
- Test.DocTest.Internal.Interpreter
- Test.DocTest.Internal.Location
- Test.DocTest.Internal.Logging
- Test.DocTest.Internal.Nix
- Test.DocTest.Internal.Options
- Test.DocTest.Internal.Parse
- Test.DocTest.Internal.Property
- Test.DocTest.Internal.Runner
- Test.DocTest.Internal.Util
- Test.DocTest
Doctest parallel: Test interactive Haskell examples
doctest-parallel
is a library that checks examples in Haddock comments. It is similar to the popular Python module with the same name.
Installation
doctest-parallel
is available from Hackage. It cannot be used as a standalone binary, rather, it expects to be integrated in a Cabal/Stack project. See examples/ for more information on how to integrate doctest-parallel
into your project.
Migrating from doctest
See issue #11 for more information.
Usage
Below is a small Haskell module. The module contains a Haddock comment with some examples of interaction. The examples demonstrate how the module is supposed to be used.
module Fib where
-- | Compute Fibonacci numbers
--
-- Examples:
--
-- >>> fib 10
-- 55
--
-- >>> fib 5
-- 5
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)
A comment line starting with >>>
denotes an expression. All comment lines following an expression denote the result of that expression. Result is defined by what a REPL (e.g. ghci) prints to stdout
and stderr
when evaluating that expression.
doctest-parallel
will fail on comments that haddock
also doesn’t like. Sometimes (e.g., #251), this means that doctest-parallel
will fail on input that GHC accepts.
Command line
Usage:
doctest [ options ]... [<module>]...
doctest --help
doctest --version
doctest --info
Options:
-jN number of threads to use
--log-level=LEVEL one of: debug, verbose, info, warning, error. Default: info.
--ghc-arg=ARG pass argument to GHC when parsing, pass multiple times for multiple flags
† --implicit-module-import import module before testing it (default)
† --randomize-order randomize order in which tests are run
† --seed=N use a specific seed to randomize test order
† --preserve-it preserve the `it` variable between examples
--nix account for Nix build environments (default)
--quiet set log level to `Error`, shorthand for `--log-level=error`
--verbose set log level to `Verbose`, shorthand for `--log-level=verbose`
--debug set log level to `Debug`, shorthand for `--log-level=debug`
--help display this help and exit
--version output version information and exit
--info output machine-readable version information and exit
Supported inverted options:
--no-nix
† --no-implicit-module-import
† --no-randomize-order (default)
† --no-preserve-it (default)
Options marked with a dagger (†) can also be used to set module level options, using
an ANN pragma like this:
{-# ANN module "doctest-parallel: --no-randomize-order" #-}
Example groups
Examples from a single Haddock comment are grouped together and share the same scope. E.g. the following works:
-- |
-- >>> let x = 23
-- >>> x + 42
-- 65
If an example fails, subsequent examples from the same group are skipped. E.g. for
-- |
-- >>> let x = 23
-- >>> let n = x + y
-- >>> print n
print n
is not tried, because let n = x + y
fails (y
is not in scope!).
Setup code
You can put setup code in a named chunk with the name $setup
.
The setup code is run before each example group. If the setup code produces
any errors/failures, all tests from that module are skipped.
Here is an example:
module Foo where
import Bar.Baz
-- $setup
-- >>> let x = 23 :: Int
-- |
-- >>> foo + x
-- 65
foo :: Int
foo = 42
Note that you should not place setup code in between the module header (module ... where
) and import declarations. GHC will not be able to parse it (issue
#167). It is best to place setup
code right after import declarations, but due to its declarative nature you can
place it anywhere in between top level declarations as well.
Multi-line input
GHCi supports commands which span multiple lines, and the same syntax works for doctest:
-- |
-- >>> :{
-- let
-- x = 1
-- y = 2
-- in x + y + multiline
-- :}
-- 6
multiline = 3
Note that >>>
can be left off for the lines following the first: this is so that
haddock does not strip leading whitespace. The expected output has whitespace
stripped relative to the :}.
Some peculiarities on the ghci side mean that whitespace at the very start is lost.
This breaks the example broken
, since the x and y aren’t aligned from ghci’s
perspective. A workaround is to avoid leading space, or add a newline such
that the indentation does not matter:
{- | >>> :{
let x = 1
y = 2
in x + y + works
:}
6
-}
works = 3
{- | >>> :{
let x = 1
y = 2
in x + y + broken
:}
3
-}
broken = 3
Multi-line output
If there are no blank lines in the output, multiple lines are handled automatically.
-- | >>> putStr "Hello\nWorld!"
-- Hello
-- World!
If however the output contains blank lines, they must be noted
explicitly with <BLANKLINE>
. For example,
import Data.List ( intercalate )
-- | Double-space a paragraph.
--
-- Examples:
--
-- >>> let s1 = "\"Every one of whom?\""
-- >>> let s2 = "\"Every one of whom do you think?\""
-- >>> let s3 = "\"I haven't any idea.\""
-- >>> let paragraph = unlines [s1,s2,s3]
-- >>> putStrLn $ doubleSpace paragraph
-- "Every one of whom?"
-- <BLANKLINE>
-- "Every one of whom do you think?"
-- <BLANKLINE>
-- "I haven't any idea."
--
doubleSpace :: String -> String
doubleSpace = (intercalate "\n\n") . lines
Matching arbitrary output
Any lines containing only three dots (...
) will match one or more lines with
arbitrary content. For instance,
-- |
-- >>> putStrLn "foo\nbar\nbaz"
-- foo
-- ...
-- baz
If a line contains three dots and additional content, the three dots will match anything within that line:
-- |
-- >>> putStrLn "foo bar baz"
-- foo ... baz
QuickCheck properties
Haddock (since version 2.13.0) has markup support for properties. Doctest can verify properties with QuickCheck. A simple property looks like this:
-- |
-- prop> \n -> abs n == abs (abs (n :: Int))
The lambda abstraction is optional and can be omitted:
-- |
-- prop> abs n == abs (abs (n :: Int))
A complete example that uses setup code is below:
module Fib where
-- $setup
-- >>> import Control.Applicative
-- >>> import Test.QuickCheck
-- >>> newtype Small = Small Int deriving Show
-- >>> instance Arbitrary Small where arbitrary = Small . (`mod` 10) <$> arbitrary
-- | Compute Fibonacci numbers
--
-- The following property holds:
--
-- prop> \(Small n) -> fib n == fib (n + 2) - fib (n + 1)
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)
If you see an error like the following, ensure that QuickCheck is a dependency of your test-suite.
<interactive>:39:3:
Not in scope: ‘polyQuickCheck’
In the splice: $(polyQuickCheck (mkName "doctest_prop"))
<interactive>:39:3:
GHC stage restriction:
‘polyQuickCheck’ is used in a top-level splice or annotation,
and must be imported, not defined locally
In the expression: polyQuickCheck (mkName "doctest_prop")
In the splice: $(polyQuickCheck (mkName "doctest_prop"))
Hiding examples from Haddock
You can put examples into named chunks, and not refer to them in the export list. That way they will not be part of the generated Haddock documentation, but Doctest will still find them.
-- $
-- >>> 1 + 1
-- 2
Using GHC extensions
You can enable GHC extensions using the following syntax:
-- >>> :set -XTupleSections
If you want to omit the information which language extensions are enabled from the Doctest examples you can use the method described in Hiding examples from Haddock, e.g.:
-- $
-- >>> :set -XTupleSections
Using GHC plugins
You can enable GHC plugins using the following syntax:
-- >>> :set -fplugin The.Plugin
Hiding Prelude
You hide the import of Prelude
by using:
-- >>> :m -Prelude
Per module options
You can override command line flags per module by using a module annotation. For example, if you know a specific module does not support test order randomization, you can disable it with:
{-# ANN module "doctest-parallel: --no-randomize-order" #-}
Test non-exposed modules
Generally, doctest-parallel
cannot test binders that are part of non-exposed modules, unless they are re-exported from exposed modules. By default doctest-parallel
will fail to do so (and report an error message), because it doesn’t track whether functions are re-exported in such a way. To test a re-exported function, add the following to the non-exposed module:
{-# ANN module "doctest-parallel: --no-implicit-module-import" #-}
This makes doctest-parallel
omit the usual module import at the start of a test.
Then, before a test -or in $setup
- add:
>>> import Exposed.Module (someFunction)
Relation to doctest
This is a fork of sol/doctest that allows running tests in parallel and aims to provide a more robust project integration method. It is not backwards compatible and expects to be setup differently. At the time of writing it has a few advantages over the base project:
- It runs tests in parallel
- It runs tests against compiled code, instead of reinterpreting your whole project
- It isolates examples in modules, ensuring your tests don’t accidentally rely on each other
- It parses cabal files to discover modules, no need for custom setup anymore!
- A minor change: it does not count lines in setup blocks as test cases
- A minor change: the testsuite has been ported to v2 commands
All in all, you can expect doctest-parallel
to run about 1 or 2 orders of magnitude faster than doctest
for large projects.
Relation to cabal-docspec
There is no direct relation between doctest-parallel
and cabal-docspec
. They are similar in some ways:
- Both projects load code from precompiled modules
- Both project aim to get rid of the need for custom setups
And different in others:
- As a fork of
doctest
,doctest-parallel
inherits the testsuitedoctest
accumulated over the years. doctest-parallel
parses Cabal project files, instead of parsing files fromdist-newstyle
. This makes it compatible with Stack, provided a.cabal
is still present.doctest-parallel
uses the GHC API to parse comments. This should in theory be more reliable (though I doubt it will ever matter in practice).doctest-parallel
runs tests in parallel.
Development
To run the tests:
cabal run spectests
cabal run doctests
Future of this project
- It would be lovely if we could get rid of the needs for
write-ghc-environment-files: always
option for Cabal. To properly do this, I think Cabal should do two things:- Deprecate GHC environment files as a way to implicitly setup environments. Instead, environment files should be written to the
dist-newstyle
directory and activated using some subcommand, e.g.cabal shell
. This avoids the many problems GHC environment files have, while retaining their functionality for people who like them. - Any subcommands should be run with
GHC_ENVIRONMENT
set - pointing to the GHC environment file. Like Stack, this would create a hassle free way of using Cabal in combination with projects/executables that use the GHC API (e.g.,clash-ghc
,doctest-parallel
).
- Deprecate GHC environment files as a way to implicitly setup environments. Instead, environment files should be written to the
- It would be nice if Cabal would expose more information by default (probably through auto-generated modules) in order for
doctest-parallel
to properly work. Specifically, it needs to know the exactdefault-extensions
,ghc-options
, andCPP
flags the project is compiled with. These options are obtainable by using a customSetup.hs
, but this has its own list of problems.- Alternatively, if comments could be included in and loaded from
.hi
files that’d solve all issues too.
- Alternatively, if comments could be included in and loaded from
- Hopefully many of the improvements made here can make their way back into
sol/doctest
.
Of course, if you wish to add a feature that’s not in this list, please feel free top open a pull request!
Contributors
- Adam Vogt
- Anders Persson
- Ankit Ahuja
- Edward Kmett
- Hiroki Hattori
- Joachim Breitner
- João Cristóvão
- Julian Arni
- Kazu Yamamoto
- Levent Erkok
- Luke Murphy
- Matvey Aksenov
- Michael Orlitzky
- Michael Snoyman
- Nick Smallbone
- Phil de Joux
- Sakari Jokinen
- Simon Hengel
- Sönke Hahn
Changes
0.3.1
- Add support for GHC 9.8
- Drop support for GHC 8.2
- Add
--ghc-arg
as a command line argument, allowing users to pass additional arguments to GHC used to parse Haddock.
0.3.0.1
- Add support for GHC 9.6
0.3.0
- Add support for Nix shell environments (#58)
Language.Haskell.GhciWrapper
has been moved toTest.DocTest.Internal.GhciWrapper
. This module was never intended to be part of the public API. (#61)- Add more elaborate debug options. You can now pass
--log-level=LEVEL
wherelevel
is one ofdebug
,verbose
,info
,warning
, orerror
. (#14)
0.2.6
getNumProcessors
is now used to detect the (default) number of GHCi subprocesses to spawn. This should more reliably use all of a system’s resources. Fixes #53.- Add Nix support. If the environment variable
NIX_BUILD_TOP
is present an extra package database is added toGHC_PACKAGE_PATH
. This isn’t expected to break existing builds, but if it does consider passing--no-nix
. (#34) - The QuickCheck example mentioned in the README now uses
abs
instead ofsort
. This prevents confusing errors whensort
is not imported. Fixes #50.
0.2.5
- Loosen Cabal bounds to >= 2.4 && < 3.9
0.2.4
- Add support for GHC 9.4 (#43)
0.2.3
- Conditionals in Cabal files are now solved (#35). Thanks to @philderbeast for the report and contributions.
- Unexpected outputs in
$setup
blocks are no longer ignored (#39)
0.2.2
- Command line arguments (such as
--randomize-order
) can now be overridden on a per-module basis (#25) - Implicit pre-test module imports can now be disabled using
--no-implicit-module-import
. This can help to test functions from non-exposed modules (#26) runModule
does not swallow import errors anymore (#28)autogen-modules
are not searched for tests anymore (#30)
0.2.1
- C include directories (Cabal field:
include-dirs
) are now passed to GHC when parsing source files (#7) - A migration guide has been added (#11)
- Test order can be randomized using
--randomize-order
. Test order can be made deterministic by adding an optional--seed=N
argument (#12) - Any non-error output can now be surpressed by
--quiet
(#20) - Doctest can now be called using a record for option passing in addition to command line arguments. See
mainFromCabalWithConfig
andmainFromLibraryWithConfig
.
0.2
Changes:
- Support for GHC 9.2 has been added (#4)
- Support for GHC 8.2 has been dropped (#3)
- The dependency
cabal-install-parsers
has been dropped. This trims the dependency tree quite a bit (#3) - The Hackage distribution now ships all files necessary to run
doctest-parallel
’s tests (Fixes #1, PR #2)
0.1
Fresh fork from sol/doctest
. See the README for an overview of all the changes.