Changelog

[0.18.0.0] - 2024-09-26

Added

• The test Assertion which displays a diff in case of error (so shouldBe, shouldReturn, golden tests and variations) will now timeout (after 2s) when computing the diff between expected and actual value. In case of timeout, the values are displayed without any diff formatting. This ensure that test suite runtime won’t be dominated by computing diff on some pathological cases.
• The smart constructor mkNotEqualButShouldHaveBeenEqual
• You can use your own diff algorithm using the constructor NotEqualButShouldHaveBeenEqualWithDiff.
• Test suite does not crash if failed assertion tries to print values containing lazy exception. For example shouldBe (1, error "nop") (2, 3) was crashing before. The exception is now reported as the failure reason for the test. Note that this can be counter intuitive, because the test is failing because values are not equal (e.g. (1, _) != (2, _)), and this will be reported differently.

Changed

The diff computation between actual value and reference changed so diff can timeout.

This does not change the usual API (shouldBe or GoldenTest), but some internal changed and you may need to adapt. The change is straightforward, most of the functions are not IO:

• stringsNotEqualButShouldHaveBeenEqual, textsNotEqualButShouldHaveBeenEqual and bytestringsNotEqualButShouldHaveBeenEqual are now IO Assertion (was Assertion) in order to implement the timeout logic described for shouldBe. The Assertion NotEqualButShouldHaveBeenEqual is removed and replaced by NotEqualButShouldHaveBeenEqualWithDiff which embed the difference between both values.
• The record field goldenTestCompare of GoldenTest changed from a -> a -> Maybe Assertion to a -> a -> IO (Maybe Assertion).

[0.17.0.2] - 2024-09-26

Changed

• Sydtest won’t crash anymore, behave weirdly, or leak resources when executed in a REPL and interrupted by C-c.

[0.17.0.1] - 2024-09-26

Changed

• Only use withArgs when the argument list isn’t already empty. This works around a concurrency issue wherein withArgs cannot be run twice from multiple threads.

[0.17.0.0] - 2024-08-04

Changed

• Allow golden tests to perform IO during comparisons

[0.16.0.0] - 2024-08-03

Changed

• opt-env-conf-based settings parsing.

[0.15.1.3] - 2024-07-20

Changed

• Fix race condition in the asynchronous runner

[0.15.1.2] - 2024-07-18

Changed

• Fix parsing filter flags so it becomes easy to select tests with spaces in their description

[0.15.1.1] - 2023-10-04

Changed

• Compatibility with optparse-applicative > 0.18.
• Compatibility with GHC >= 9.7.
• Refactored out fast-myers-diff into its own package.

[0.15.1.0] - 2023-07-28

Added

• setupAroundWithAll: so it’s easier to use multiple outer resources to provide an inner resource, without the need of extra type annotation.

[0.15.0.0] - 2023-04-08

Added

• DefBeforeAllWithNode: so that beforeAllWith can be defined in terms of it and have better parallelism properties.
• DefSetupNode: so that beforeAll_ can be defined in terms of it and have better parallelism properties.

[0.14.0.0] - 2023-04-05

Added

• Profiling mode, for figuring out why your test suite is slow. Use --profile to turn it on.

Changed

• An improved asynchronous test runner.
• Made --debug imply --retries 0

[0.13.0.4] - 2023-03-31

Added

• Allow supplying multiple --filter/--match flags.
• Add short names -m and -f for the filter flag options.

[0.13.0.3] - 2023-03-27

Added

• Compatibility with mtl-2.3

[0.13.0.2] - 2023-03-24

Changed

• Improved white space diffing

[0.13.0.1] - 2023-01-13

Changed

• Replaced the diffing algorithm by a custom one. This has sped up diffing at least 100x and let us get rid of the Diff and split dependencies.
• Reworded ‘likely not flaky’ to ‘does not look flaky’ to be more technically accurate as we know nothing about the likelihood of flakiness.

[0.13.0.0] - 2022-10-14

Changed

• Changed the default of –golden-start to false. If this causes any of your tests to fail, that means that you are not correctly shipping the golden results along with your test suite.
• Fixed a typo in the parallelisation warning that you see when you run sydtest in parallel with only one core..

[0.12.0.1] - 2022-10-06

• Released the changelog with a date.

[0.12.0.0] - 2022-10-06

Added

• Automatic flakiness diagnostics, see the README This adds the potentiallyFlaky and potentiallyFlakyWith functions.
• Test.Syd now also exports pPrint.
• The modifyRetries, withoutRetries, withRetries functions, to allow configuration of the number of retries independently of whether flakiness is allowed.
• The TestRunReport type, so that a ResultForest contains information about all runs of a test instead of only the last.
• Expectation of failure. This adds the expectFailing, expectPassing, and withExpectationMode functions.

Changed

• Fixed: Fail-fast now works correctly together with fail-on-flaky
• Fixed that flags with a no- prefix did not parse correctly and could therefore not be used.
• The FlakinessMode type no longer contains a number of retries. The number of retries is now configured separately.
• Fixed that xdescribe would only result in one pending test instead of the same number of tests as are marked as pending.
• Fixed that specify and prop would show a callstack from inside sydtest instead of from where you used them.
• Sydtest now sets the global pseudorandomness seed in the random library to the given seed using setStdGen for extra reproducability.

Removed

[0.11.0.2] - 2022-09-7

Changed

• Added the shouldMatchList expectation, corresponding the one from hspec-expectations.

[0.11.0.1] - 2022-06-28

Changed

• Made the type of setupAround more general.

[0.11.0.0] - 2022-06-28

Changed

• printOutputSpecForest now uses Data.Text.Lazy.IO to output the report, so it will try to respect the system’s locale.
• renderResultReport now outputs a Text.Builder instead of a ByteString.Builder and a newline at the end.

[0.10.1.0] - 2022-06-04

Changed

• Fixed a bug where multiple layers of contextual failures were not unwrapped and shown correctly.
• Made ‘Assertion’ fields strict.

[0.10.0.0] - 2022-04-28

Added

• Allow supplying filters without a flag, so hello instead of --filter hello or --match hello.
• Simplified the way exceptions work to allow for attaching context to any failure instead of only Assertions.

[0.9.0.0] - 2022-04-13

Added

• Optional type-safe progress reporting for property tests.

[0.8.0.1] - 2022-02-11

Changed

• Fixed property tests without labels still showing labels in the output.

[0.8.0.0] - 2022-02-11

Changed

• The TestDefM now contains a TestDefEnv which also contains the test description path, along with TestRunSettings.
• Removed the MonadState () instance of TestDefM. It was just silly.
• Changed the internals of TestDefM to use ReaderT and a strict WriterT instead of RWST.
• Renamed wrapRWST to wrapForest.
• Fixed the property label output to use the right total.
• Moved filterTestForest and randomiseTestForest from Test.Syd.TestDefM to Test.Syd.SpecDef, where SpecDefForest is defined.

Added

• getTestDescriptionPath to get the test description path upwards from inside a test definition.

[0.7.0.1] - 2021-12-23

Changed

• Fixed shouldStartWith to test on the prefix rather then infix

[0.7.0.0] - 2021-12-15

Added

• Flaky tests now show up in the failure report when --fail-on-flaky is active.
• Flakiness information like the number of retries is now shown in the failure report for real (non-flaky) failures.

Changed

• Simplified the way settings are passed around.

[0.6.1.0] - 2021-12-10

Added

• The flakyWith combinator, which is like flaky, but lets you also add a message to your team.

[0.6.0.0] - 2021-11-12

Changed

• Started using autodocodec instead of yamlparse-applicative.

[0.5.0.0] - 2021-11-12

Added

• The flakiness combinators (flaky, notFlaky, and withFlakiness) to mark a test group as potentially flaky.
• The --fail-on-flaky flag to falsify flakiness.
• Experimental Windows support

Changed

• Fixed the interpretation of max-size vs max-success in the configuration file and environment parsing.

[0.4.1.0] - 2021-10-10

Added

• The --random-seed option to use random seeds instead of the fixed seed that is used by default.

[0.4.0.0] - 2021-09-02

Added

• The --debug option.

Changed

• Redid the entire flags parsing. This should be backward compatible, and result in a nicer --help overview.

[0.3.0.3] - 2021-08-07

Changed

• Show the total number of examples in the output as well

[0.3.0.2] - 2021-07-06

Changed

• Accept options using American spelling as well.

[0.3.0.1] - 2021-06-20

Changed

• Turned off shrinking when using around and friends. See https://github.com/nick8325/quickcheck/issues/331.

[0.3.0.0] - 2021-06-17

Added

• An IsTest (ReaderT env IO a) instance.

Deleted

• Test.Syd.Def.Env, which contained eit and withTestEnv Now that ReaderT env IO a is also in IsTest, you can just use it for this.

[0.2.0.0] - 2021-06-03

Added

• beforeWith and beforeWith'
• scenarioDir and scenarioDirRecur for scenario testing.
• bracketSetupFunc

Changed

• The SetupFunc has been simplified to only take one type parameter.

Deleted

• composeSetupFunc, now obsolete: use <=< instead.
• connectSetupFunc, now obsolete: use >=> instead.
• wrapSetupFunc, now entirely obsolete.
• unwrapSetupFunc, now entirely obsolete.
• makeSimpleSetupFunc, now obsolete: Use the SetupFunc constructor directly.
• useSimpleSetupFunc, now obsolete: Use the unSetupFunc function directly.

[0.1.0.0] - 2021-03-07

Various fixes

[0.0.0.0] - 2020-12-26

Initial release