tasty
Modern and extensible testing framework
https://github.com/UnkindPartition/tasty
Version on this page: | 1.4.3 |
LTS Haskell 22.33: | 1.4.3 |
Stackage Nightly 2024-09-07: | 1.5@rev:2 |
Latest on Hackage: | 1.5.1@rev:1 |
tasty-1.4.3@sha256:5b53fded93109f6704b599c3498eb73c5b0ed1a975f1912dd98b4b19c61f8bc9,2653
Module documentation for 1.4.3
- Test
- Test.Tasty
- Test.Tasty.Ingredients
- Test.Tasty.Options
- Test.Tasty.Patterns
- Test.Tasty.Providers
- Test.Tasty.Runners
- Test.Tasty
Tasty
Tasty is a modern testing framework for Haskell.
It lets you combine your unit tests, golden tests, QuickCheck/SmallCheck properties, and any other types of tests into a single test suite.
Features:
- Run tests in parallel but report results in a deterministic order
- Filter the tests to be run using patterns specified on the command line
- Hierarchical, colored display of test results
- Reporting of test statistics
- Acquire and release resources (sockets, temporary files etc.) that can be shared among several tests
- Extensibility: add your own test providers and ingredients (runners) above and beyond those provided
To find out what’s new, read the change log.
Example
Here’s how your test.hs
might look like:
import Test.Tasty
import Test.Tasty.SmallCheck as SC
import Test.Tasty.QuickCheck as QC
import Test.Tasty.HUnit
import Data.List
import Data.Ord
main = defaultMain tests
tests :: TestTree
tests = testGroup "Tests" [properties, unitTests]
properties :: TestTree
properties = testGroup "Properties" [scProps, qcProps]
scProps = testGroup "(checked by SmallCheck)"
[ SC.testProperty "sort == sort . reverse" $
\list -> sort (list :: [Int]) == sort (reverse list)
, SC.testProperty "Fermat's little theorem" $
\x -> ((x :: Integer)^7 - x) `mod` 7 == 0
-- the following property does not hold
, SC.testProperty "Fermat's last theorem" $
\x y z n ->
(n :: Integer) >= 3 SC.==> x^n + y^n /= (z^n :: Integer)
]
qcProps = testGroup "(checked by QuickCheck)"
[ QC.testProperty "sort == sort . reverse" $
\list -> sort (list :: [Int]) == sort (reverse list)
, QC.testProperty "Fermat's little theorem" $
\x -> ((x :: Integer)^7 - x) `mod` 7 == 0
-- the following property does not hold
, QC.testProperty "Fermat's last theorem" $
\x y z n ->
(n :: Integer) >= 3 QC.==> x^n + y^n /= (z^n :: Integer)
]
unitTests = testGroup "Unit tests"
[ testCase "List comparison (different length)" $
[1, 2, 3] `compare` [1,2] @?= GT
-- the following test does not hold
, testCase "List comparison (same length)" $
[1, 2, 3] `compare` [1,2,2] @?= LT
]
And here is the output of the above program:
(Note that whether QuickCheck finds a counterexample to the third property is determined by chance.)
Packages
tasty is the core package. It contains basic definitions and APIs and a console runner.
In order to create a test suite, you also need to install one or more «providers» (see below).
Providers
The following providers exist:
- tasty-hunit — for unit tests (based on HUnit)
- tasty-golden — for golden tests, which are unit tests whose results are kept in files
- tasty-smallcheck — exhaustive property-based testing (based on smallcheck)
- tasty-quickcheck — for randomized property-based testing (based on QuickCheck)
- tasty-hedgehog — for randomized property-based testing (based on Hedgehog)
- tasty-hspec — for Hspec tests
- tasty-leancheck — for enumerative property-based testing (based on LeanCheck)
- tasty-program — run external program and test whether it terminates successfully
- tasty-wai — for testing wai endpoints.
- tasty-inspection-testing — for compile-time testing of code properties (based on inspection-testing).
It’s easy to create custom providers using the API from Test.Tasty.Providers
.
Ingredients
Ingredients represent different actions that you can perform on your test suite. One obvious ingredient that you want to include is one that runs tests and reports the progress and results.
Another standard ingredient is one that simply prints the names of all tests.
It is possible to write custom ingredients using the API from Test.Tasty.Runners
.
Some ingredients that can enhance your test suite are:
- tasty-ant-xml adds a possibility to write the test results in a machine-readable XML format, which is understood by various CI systems and IDEs
- tasty-rerun adds support for minimal test reruns by recording previous test runs and using this information to filter the test tree. For example, you can use this ingredient to only run failed tests, or only run tests that threw an exception.
- tasty-html adds the possibility to write the test results as a HTML file
- tasty-stats adds the possibility to collect statistics of the test suite in a CSV file.
Other packages
- tasty-th automatically discovers tests based on the function names and generate the boilerplate code for you
- tasty-hunit-adapter converts existing HUnit test suites into tasty test suites
- tasty-discover automatically discovers your tests.
- tasty-expected-failure provides test markers for when you expect failures or wish to ignore tests.
- tasty-bench covers performance
regression testing and extends
tasty
to a benchmark framework similar tocriterion
andgauge
.
Options
Options allow one to customize the run-time behavior of the test suite, such as:
- mode of operation (run tests, list tests, run tests quietly etc.)
- which tests are run (see «Patterns» below)
- parameters of individual providers (like depth of search for SmallCheck)
Setting options
There are two main ways to set options:
Runtime
When using the standard console runner, the options can be passed on the
command line or via environment variables. To see the available options, run
your test suite with the --help
flag. The output will look something like this
(depending on which ingredients and providers the test suite uses):
% ./test --help
Mmm... tasty test suite
Usage: test [-p|--pattern PATTERN] [-t|--timeout DURATION] [-l|--list-tests]
[-j|--num-threads NUMBER] [-q|--quiet] [--hide-successes]
[--color never|always|auto] [--ansi-tricks ARG]
[--smallcheck-depth NUMBER] [--smallcheck-max-count NUMBER]
[--quickcheck-tests NUMBER] [--quickcheck-replay SEED]
[--quickcheck-show-replay] [--quickcheck-max-size NUMBER]
[--quickcheck-max-ratio NUMBER] [--quickcheck-verbose]
[--quickcheck-shrinks NUMBER]
Available options:
-h,--help Show this help text
-p,--pattern PATTERN Select only tests which satisfy a pattern or awk
expression
-t,--timeout DURATION Timeout for individual tests (suffixes: ms,s,m,h;
default: s)
-l,--list-tests Do not run the tests; just print their names
-j,--num-threads NUMBER Number of threads to use for tests execution
(default: # of cores/capabilities)
-q,--quiet Do not produce any output; indicate success only by
the exit code
--hide-successes Do not print tests that passed successfully
--color never|always|auto
When to use colored output (default: auto)
--ansi-tricks ARG Enable various ANSI terminal tricks. Can be set to
'true' or 'false'. (default: true)
--smallcheck-depth NUMBER
Depth to use for smallcheck tests
--smallcheck-max-count NUMBER
Maximum smallcheck test count
--quickcheck-tests NUMBER
Number of test cases for QuickCheck to generate.
Underscores accepted: e.g. 10_000_000
--quickcheck-replay SEED Random seed to use for replaying a previous test run
(use same --quickcheck-max-size)
--quickcheck-show-replay Show a replay token for replaying tests
--quickcheck-max-size NUMBER
Size of the biggest test cases quickcheck generates
--quickcheck-max-ratio NUMBER
Maximum number of discared tests per successful test
before giving up
--quickcheck-verbose Show the generated test cases
--quickcheck-shrinks NUMBER
Number of shrinks allowed before QuickCheck will fail
a test
Every option can be passed via environment. To obtain the environment variable
name from the option name, replace hyphens -
with underscores _
, capitalize
all letters, and prepend TASTY_
. For example, the environment equivalent of
--smallcheck-depth
is TASTY_SMALLCHECK_DEPTH
.
Note on boolean options: by convention, boolean (“on/off”) options are specified
using a switch on the command line, for example --quickcheck-show-replay
instead of --quickcheck-show-replay=true
. However, when
passed via the environment, the option value needs to be True
or False
(case-insensitive), e.g. TASTY_QUICKCHECK_SHOW_REPLAY=true
.
If you’re using a non-console runner, please refer to its documentation to find out how to configure options during the run time.
Compile-time
You can also specify options in the test suite itself, using
localOption
. It can be applied not only to the whole test tree, but also to
individual tests or subgroups, so that different tests can be run with
different options.
It is possible to combine run-time and compile-time options, too, by using
adjustOption
. For example, make the overall testing depth configurable
during the run time, but increase or decrease it slightly for individual
tests.
This method currently doesn’t work for ingredient options, such as --quiet
or
--num-threads
. You can set them by setting the corresponding environment
variable before calling defaultMain
:
import Test.Tasty
import System.Environment
main = do
setEnv "TASTY_NUM_THREADS" "1"
defaultMain _
Patterns
It is possible to restrict the set of executed tests using the -p/--pattern
option.
Tasty patterns are very powerful, but if you just want to quickly run tests containing foo
somewhere in their name or in the name of an enclosing test group, you can just
pass -p foo
. If you need more power, or if that didn’t work as expected, read
on.
A pattern is an awk expression. When the expression is evaluated, the field $1
is set to the outermost test group name, $2
is set to the next test group
name, and so on up to $NF
, which is set to the test’s own name. The field $0
is set to all other fields concatenated using .
as a separator.
As an example, consider a test inside two test groups:
testGroup "One" [ testGroup "Two" [ testCase "Three" _ ] ]
When a pattern is evaluated for the above test case, the available fields and variables are:
$0 = "One.Two.Three"
$1 = "One"
$2 = "Two"
$3 = "Three"
NF = 3
Here are some examples of awk expressions accepted as patterns:
$2 == "Two"
— select the subgroupTwo
$2 == "Two" && $3 == "Three"
— select the test or subgroup namedThree
in the subgroup namedTwo
$2 == "Two" || $2 == "Twenty-two"
— select two subgroups$0 !~ /skip/
or! /skip/
— select tests whose full names (including group names) do not contain the wordskip
$NF !~ /skip/
— select tests whose own names (but not group names) do not contain the wordskip
$(NF-1) ~ /QuickCheck/
— select tests whose immediate parent group name containsQuickCheck
As an extension to the awk expression language, if a pattern pat
contains only
letters, digits, and characters from the set ._ -
(period, underscore, space, hyphen),
it is treated like /pat/
(and therefore matched against $0
).
This is so that we can use -p foo
as a shortcut for -p /foo/
.
The only deviation from awk that you will likely notice is that Tasty
does not implement regular expression matching.
Instead, $1 ~ /foo/
means that the string foo
occurs somewhere in $1
,
case-sensitively. We want to avoid a heavy dependency of regex-tdfa
or
similar libraries; however, if there is demand, regular expression support could
be added under a cabal flag.
The following operators are supported (in the order of decreasing precedence):
The following built-in functions are supported:
substr(s, m[, n])
Return the at most n
-character substring of s
that begins at
position m
, numbering from 1. If n
is omitted, or if n
specifies
more characters than are left in the string, the length of the substring
will be limited by the length of the string s
.
tolower(s)
Convert the string s
to lower case.
toupper(s)
Convert the string s
to upper case.
match(s, pat)
Return the position, in characters, numbering from 1, in string s
where the
pattern pat
occurs, or zero if it does not occur at all.
pat
must be a literal, not an expression, e.g. /foo/
.
length([s])
Return the length, in characters, of its argument taken as a string, or of the whole record, $0
, if there is no argument.
Running tests in parallel
In order to run tests in parallel, you have to do the following:
- Compile (or, more precisely, link) your test program with the
-threaded
flag; - Launch the program with
+RTS -N -RTS
.
Timeout
To apply timeout to individual tests, use the --timeout
(or -t
) command-line
option, or set the option in your test suite using the mkTimeout
function.
Timeouts can be fractional, and can be optionally followed by a suffix ms
(milliseconds), s
(seconds), m
(minutes), or h
(hours). When there’s no
suffix, seconds are assumed.
Example:
./test --timeout=0.5m
sets a 30 seconds timeout for each individual test.
Options controlling console output
The following options control behavior of the standard console interface:
Custom options
It is possible to add custom options, too.
To do that,
- Define a datatype to represent the option, and make it an instance of
IsOption
- Register the options with the
includingOptions
ingredient - To query the option value, use
askOption
.
See the Custom options in Tasty article for some examples.
Project organization and integration with Cabal
There may be several ways to organize your project. What follows is not Tasty’s requirements but my recommendations.
Tests for a library
Place your test suite sources in a dedicated subdirectory (called tests
here) instead of putting them among the main library sources.
The directory structure will be as follows:
my-project/
my-project.cabal
src/
...
tests/
test.hs
Mod1.hs
Mod2.hs
...
test.hs
is where your main
function is defined. The tests may be
contained in test.hs
or spread across multiple modules (Mod1.hs
, Mod2.hs
,
…) which are then imported by test.hs
.
Add the following section to the cabal file (my-project.cabal
):
test-suite test
default-language:
Haskell2010
type:
exitcode-stdio-1.0
hs-source-dirs:
tests
main-is:
test.hs
build-depends:
base >= 4 && < 5
, tasty >= 0.7 -- insert the current version here
, my-project -- depend on the library we're testing
, ...
Tests for a program
All the above applies, except you can’t depend on the library if there’s no library. You have two options:
- Re-organize the project into a library and a program, so that both the program and the test suite depend on this new library. The library can be declared in the same cabal file.
- Add your program sources directory to the
Hs-source-dirs
. Note that this will lead to double compilation (once for the program and once for the test suite).
Dependencies
Tasty executes tests in parallel to make them finish faster. If this parallelism is not desirable, you can declare dependencies between tests, so that one test will not start until certain other tests finish.
Dependencies are declared using the after
combinator:
after AllFinish "pattern" my_tests
will execute the test treemy_tests
only after all tests that match the pattern finish.after AllSucceed "pattern" my_tests
will execute the test treemy_tests
only after all tests that match the pattern finish and only if they all succeed. If at least one dependency fails, thenmy_tests
will be skipped.
The relevant types are:
after
:: DependencyType -- ^ whether to run the tests even if some of the dependencies fail
-> String -- ^ the pattern
-> TestTree -- ^ the subtree that depends on other tests
-> TestTree -- ^ the subtree annotated with dependency information
data DependencyType = AllSucceed | AllFinish
The pattern follows the same AWK-like syntax and semantics as described in
Patterns. There is also a variant named after_
that accepts the
AST of the pattern instead of a textual representation.
Let’s consider some typical examples. (A note about terminology: here
by “resource” I mean anything stateful and external to the test: it could be a file,
a database record, or even a value stored in an IORef
that’s shared among
tests. The resource may or may not be managed by withResource
.)
-
Two tests, Test A and Test B, access the same shared resource and cannot be run concurrently. To achieve this, make Test A a dependency of Test B:
testGroup "Tests accessing the same resource" [ testCase "Test A" $ ... , after AllFinish "Test A" $ testCase "Test B" $ ... ]
-
Test A creates a resource and Test B uses that resource. Like above, we make Test A a dependency of Test B, except now we don’t want to run Test B if Test A failed because the resource may not have been set up properly. So we use
AllSucceed
instead ofAllFinish
testGroup "Tests creating and using a resource" [ testCase "Test A" $ ... , after AllSucceed "Test A" $ testCase "Test B" $ ... ]
Here are some caveats to keep in mind regarding dependencies in Tasty:
-
If Test B depends on Test A, remember that either of them may be filtered out using the
--pattern
option. Collecting the dependency info happens after filtering. Therefore, if Test A is filtered out, Test B will run unconditionally, and if Test B is filtered out, it simply won’t run. -
Tasty does not currently check whether the pattern in a dependency matches anything at all, so make sure your patterns are correct and do not contain typos. Fortunately, misspecified dependencies usually lead to test failures and so can be detected that way.
-
Dependencies shouldn’t form a cycle, otherwise Tasty with fail with the message “Test dependencies form a loop.” A common cause of this is a test matching its own dependency pattern.
-
Using dependencies may introduce quadratic complexity. Specifically, resolving dependencies is O(number_of_tests × number_of_dependencies), since each pattern has to be matched against each test name. As a guideline, if you have up to 1000 tests, the overhead will be negligible, but if you have thousands of tests or more, then you probably shouldn’t have more than a few dependencies.
Additionally, it is recommended that the dependencies follow the natural order of tests, i.e. that the later tests in the test tree depend on the earlier ones and not vice versa. If the execution order mandated by the dependencies is sufficiently different from the natural order of tests in the test tree, searching for the next test to execute may also have an overhead quadratic in the number of tests.
FAQ
-
Q: When my tests write to stdout/stderr, the output is garbled. Why is that and what do I do?
A: It is not recommended that you print anything to the console when using the console test reporter (which is the default one). See #103 for the discussion.
Some ideas on how to work around this:
- Use testCaseSteps (for tasty-hunit only).
- Use a test reporter that does not print to the console (like tasty-ant-xml).
- Write your output to files instead.
-
Q: Why doesn’t the
--hide-successes
option work properly? The test headings show up and/or the output appears garbled.A: This can happen sometimes when the terminal is narrower than the output. A workaround is to disable ANSI tricks: pass
--ansi-tricks=false
on the command line or setTASTY_ANSI_TRICKS=false
in the environment.See issue #152.
-
Q: Patterns with slashes do not work on Windows. How can I fix it?
A: If you are running Git for Windows terminal, it has a habit of converting slashes to backslashes. Set
MSYS_NO_PATHCONV=1
to prevent this behaviour, or follow other suggestions from Known Issues.
Press
Blog posts and other publications related to tasty. If you wrote or just found something not mentioned here, send a pull request!
- Holy Haskell Project Starter
- First time testing, also with FP Complete (tasty has been added to stackage since then)
- 24 Days of Hackage: tasty
- Resources in Tasty
- Custom options in Tasty
- Resources in Tasty (update)
- Announcing tasty-rerun
- Code testing in Haskell revisited (with Tasty)
- New patterns in tasty
- Screencast: Dynamic Test Suites in Haskell using Hspec and Tasty
- Automatically generated directories for tasty tests
GHC version support policy
We only support the GHC/base versions from the last 5 years.
Maintainers
Roman Cheplyaka is the primary maintainer.
Oliver Charles is the backup maintainer. Please get in touch with him if the primary maintainer cannot be reached.
Changes
Changes
Version 1.4.3
2022-12-18
- Drop
unbounded-delays
dependency on 64-bit machines (#344). - Drop
clock
dependency, decomissionclock
flag (#345). - Drop
wcwidth
dependency (#346) - Tested with GHC 8.0 - 9.4.
Version 1.4.2.3
2022-05-10
- Drop
mtl
dependency - Warning-free under GHC 8.0 - 9.2
Version 1.4.2.2
2022-05-10
- Fix compilation with
mtl-2.3
- Tested with GHC 8.0 - 9.2; dropped support for GHC 7.x
Version 1.4.2.1
Fix warnings under GHC 9.2
Version 1.4.2
- Add
consoleTestReporterWithHook
- Suggest pattern to rerun an individual failing test
- Add
Test.Tasty.Patterns.Printer
Version 1.4.1
Deduplicate command line options when there is more than one TestReporter.
Expose the function that does that, uniqueOptionDescriptions
.
Version 1.4.0.3
Fix CPP warning/error macro expansion producing 'defined' has undefined behavior
Version 1.4.0.2
Automatically disable the dependency on the clock
package when compiled by ghcjs.
Version 1.4.0.1
The only point of this release is to introduce compatibility with GHCs back to 7.0 (see https://github.com/UnkindPartition/tasty/pull/287).
Note, however, that these changes are not merged to the master branch, and the future releases will only support the GHC/base versions from the last 5 years, as per our usual policy. To test with even older GHCs, you’ll have to use this particular version of tasty (or have the constraint solver pick it for you when testing with older GHCs).
The source of this release is in the support-old-ghcs
branch of the tasty
repository.
Version 1.4
- Change the
TreeFold
data type to give all functions access toOptionSet
- Fix a bug where a looping failure message escaped the time out set for the test
- Fix a bug where pattern changes inside the
TestTree
weren’t respected
Version 1.3.1
- Add an ability for a test provider to print colorful/formatted output
Version 1.3
IsOption
has a new methodshowDefaultValue
for customizing howdefaultValue
s are rendered in the--help
output.- Drop support for GHCs older than 5 years
- Do not install handlers for the signals that dump core
- Export the
AnsiTricks
type/option - In addition to a
Parser
,optionParser
andsuiteOptionParser
now return a[String]
representing warning messages:- A warning is emitted if an
IsOption
instance defines multiple options in the implementation ofoptionCLParser
. - An warning is emitted if an
IsOption
instance’soptionCLParser
implementation assigns a default value (e.g., withOptions.Applicative.value
), as this interferes withtasty
’s ability to read environment variable arguments.
- A warning is emitted if an
Version 1.2.3
- Expose
computeStatistics
fromTest.Tasty.Ingredients.ConsoleReporter
. - Ensure that
finally
andbracket
work as expected inside tests when the test suite is interrupted by Ctrl-C.
Version 1.2.2
- Expose timed and getTime
- Add parseOptions
- Allow to disable ANSI tricks with –ansi-tricks=false
Version 1.2.1
- Document and expose installSignalHandlers
- Enable colors in Emacs and other almost-ANSI-capable terminals
Version 1.2
Make it possible to declare dependencies between tests (see the README for details)
Version 1.1.0.4
Make tasty work with GHCJS
Version 1.1.0.3
Fix compatibility with GHC 8.6
Version 1.1.0.2
Fix a bug where some (mostly Asian) characters would break alignment in the terminal output
Version 1.1.0.1
Fix a bug where -l
was still using /
instead of .
as a field separator
Version 1.1
NOTE: This major release contains some breaking changes to the semantics of patterns.
In the original pattern design I didn’t notice the conflict between using /
as
a field separator and as the AWK syntax for pattern matching /.../
.
The new patterns have been around for a relatively short time (5 months), so hopefully the breakage won’t be too big. I’m sorry about any problems caused by the change.
See https://github.com/UnkindPartition/tasty/issues/220 for the discussion.
-
The field separator in patterns is changed from slash (
/
) to period (.
), and.
is now allowed in raw patterns.The field separator is used to join the group names and the test name when comparing to a pattern such as
-p foo
or-p /foo/
.If you used
-p 'foo/bar'
or
-p '/foo\/bar/'
before, now you should use
-p 'foo.bar'
or
-p '/foo.bar/'
if you meant “test/group
bar
inside groupfoo
, or-p '/foo\/bar/'
if you meant “test/group containing
foo/bar
in the name”.The need for escaping the slash inside the
/.../
pattern was precisely the motivation for this change. -
Raw patterns (ones that are not AWK expressions) may no longer contain slashes (
/
).So
-p 'foo/bar'
is no longer allowed, and
-p '/foo/'
is now parsed as an AWK expression
/foo/
, whereas before it was treated as a raw pattern and converted to/\/foo\//
.The reason for this change is that
/foo/
is a valid AWK expression and should be parsed as such. -
Raw patterns may now contain hyphens, so e.g.
-p type-checking
now works.In theory this makes some valid AWK expressions (such as
NF-2
) not to be parsed as such, but they are either unlikely to be useful or could also be expressed in other ways (NF!=2
). -
Several new exports, mostly for testing/debugging patterns:
TestPattern
now has aShow
instance;TestPattern
andExpr
now haveEq
instances.- The constructors of
TestPattern
are now exported. parseAwkExpr
is introduced and can be used in ghci to see how an AWK expression is parsed. (For parsing test patterns, which include raw patterns in addition to AWK expression, useparseTestPattern
.)
Version 1.0.1.1
Fix a bug where a test suite that uses resources would hang if interrupted
Version 1.0.1
- Add a
safeReadBool
function, for case-insensitive parsing of boolean options - Convert all tasty’s own options to case-insensitive
Version 1.0.0.1
Adjust lower bounds for the dependencies (mtl and optparse-applicative)
Version 1.0
- New pattern language (see the README and/or the blog post)
- Make the
clock
dependency optional
Version 0.12.0.1
Fix compatibility with GHC 8.4
Version 0.12
Backward compat breaking revision of Test.Tasty.Ingredients.ConsoleReporter
that exposes the name of tests/groups.
Version 0.11.3
Expose and document several of the internals of
Test.Tasty.Ingredients.ConsoleReporter
.
Version 0.11.2.5
Fix compatibility with GHC 7.4
Version 0.11.2.4
- Make the
--quiet
mode more efficient on a large number of tests - Fix a bug where a cursor would disappear if the test suite was terminated by a signal other than SIGINT.
Version 0.11.2.3
Make filtering tests (-p
) work faster
Version 0.11.2.2
Fix a critical bug in the quiet mode (-q
/--quiet
):
the exit status could be wrong or the test suite could hang.
Version 0.11.2.1
Fix compatibility with the latest unbounded-delays
Version 0.11.2
Add composeReporters
, a function to run multiple reporter ingredients
Version 0.11.1
Introduce mkOptionCLParser
and mkFlagCLParser
Version 0.11.0.4
Fix compatibility with optparse-applicative-0.13
Version 0.11.0.3
Switch from regex-tdfa-rc
to regex-tdfa
, which got a new maintainer.
Version 0.11.0.2
Clarify IsTest
’s specification with regard to exceptions
Version 0.11.0.1
Use monotonic clock when measuring durations.
Version 0.11
New field resultShortDescription
of Result
Version 0.10.1.2
- Improve the docs
- Fix compatibility with GHC HEAD
Version 0.10.1.1
- Prevent parsing non-positive number of threads via program options (#104)
- Buffer output to avoid slowdowns when printing test results (#101)
- Default to using the maximum number of available cores for test execution
Version 0.10.1
Export Test.Tasty.Runners.formatMessage
Version 0.10.0.4
Don’t output ANSI codes for the Emacs terminal emulator
Version 0.10.0.3
Better handle the situation when there are no ingredients to run
Version 0.10.0.2
Split the changelog into per-project changelogs
Version 0.10.0.1
Update to optparse-applicative 0.11
Version 0.10
- Add the
--color
option - Timings
- Introduce the
Time
type synonym - Change the types of
launchTestTree
andTestReporter
to accept the total run time consoleTestReporter
now displays the timings
- Introduce the
Version 0.9.0.1
Upgrade to optparse-applicative-0.10.
Version 0.8.1.3
Be careful not to export the Show (a -> b)
instance, see
https://github.com/UnkindPartition/tasty/issues/71
Version 0.8.1.2
Hide cursor when running tests
Version 0.8.1.1
Fix for GHC 7.9
Version 0.8.0.4
Remove the old ‘colors’ flag description from the cabal file
Version 0.8.0.2
Make ansi-terminal an unconditional dependency
Version 0.8
Test.Tasty.Ingredients
is now exposedTest.Tasty.Ingredients.Basic
is added, which exports the ingredients defined in thetasty
package. These exports should now be used instead of ones exported fromTest.Tasty.Runners
- The
Result
type is now structured a bit differently. Providers now should usetestPassed
andtestFailed
functions instead of constructingResult
s directly. - Add «quiet mode» (see README)
- Add «hide successes» mode (see README)
- Add short command-line options:
-j
for--num-threads
,-p
for--pattern
- Add timeout support
AppMonoid
is renamed toTraversal
for consistency with the ‘reducers’ package. Another similar wrapper,Ap
, is introduced.- Fix a resources bug (resources were not released if the test suite was interrupted)
- The type of
launchTestTree
is changed. It now takes a continuation as an argument. This is necessary to fix the bug mentioned above. - Add
flagCLParser
to be used as theoptionCLParser
implementation for boolean options. - Add the ability to pass options via environment
Version 0.7
- Use
regex-tdfa
instead ofregex-posix
(which is a native implementation, and as such is more portable) foldTestTree
now takes the algebra in the form of a record rather than multiple arguments, to minimize breakage when new nodes are added or existing ones changewithResource
now passes the IO action to get the resource to the inner test tree
Version 0.6
- Better handling of exceptions that arise during resource creation or disposal
- Expose the
AppMonoid
wrapper - Add
askOption
andinludingOptions
Version 0.5.2.1
Depend on ansi-terminal >= 0.6.1. This fixes some issues with colors on Windows.
Version 0.5.2
- Export
Result
andProgress
fromTest.Tasty.Runners
- Make it clear that only GHC 7.4+ is supported
Version 0.5.1
Export ResourceSpec
from Test.Tasty.Runners
Version 0.5
Add a capability to acquire and release resources. See the «Resources» section
in the Test.Tasty
docs.
For the end users, the API is backwards-compatible.
Test runners may have to be adjusted — there is a new constructor of TestTree
and a new argument of foldTestTree
.
Version 0.4.2
Add defaultIngredients
Version 0.4.1.1
Print the failure description in red
Version 0.4.0.1
Fix a bug (#25)
Version 0.4
The big change in this release is introduction of ingredients, which is a replacement for runners. But unless you have a custom runner, this is unlikely to affect you much.
The Ingredient
data type has replaced the Runner
type.
The following functions have been renamed and possibly changed their types:
defaultMainWithRunner
→defaultMainWithIngredients
treeOptionParser
→suiteOptionParser
getTreeOptions
→treeOptions
runUI
→consoleTestReporter
Added in this release:
suiteOptions
optionParser
- functions operating on ingredients
testsNames
- the
listingTests
ingredient and its option,ListTests
NumThreads
is no longer a core option, but is automatically included in the
test reporting ingredients (see its haddock).
Version 0.3.1
- Proper reporting of (some) non-terminating tests (#15)
- Upgrade to optparse-applicative 0.6
Version 0.3
- Restrict dependency versions
- Fix a bug where non-terminating test would lead to a deadlock (#15)
Version 0.2
- Add an
execRunner
function - Make
Runner
returnIO Bool
Version 0.1.1
Set lower bound on optparse-applicative dependency version