criterion
Robust, reliable performance measurement and analysis
http://www.serpentine.com/criterion
Version on this page: | 1.6.3.0@rev:1 |
LTS Haskell 22.39: | 1.6.3.0@rev:1 |
Stackage Nightly 2024-10-31: | 1.6.4.0 |
Latest on Hackage: | 1.6.4.0 |
criterion-1.6.3.0@sha256:62f9196ded68c39855a3a5dadb35dd43828ca091eeeb2ff38fdd259040d8e694,5616
Module documentation for 1.6.3.0
Criterion: robust, reliable performance measurement
This package provides the Criterion module, a Haskell library for measuring and analysing software performance.
To get started, read the online tutorial, and take a look at the programs in the examples directory.
Get involved!
Please report bugs via the github issue tracker.
Master github repository:
git clone https://github.com/haskell/criterion.git
Authors
This library is written and maintained by Bryan O’Sullivan, [email protected].
Changes
1.6.3.0
- Remove a use of the partial
head
function withincriterion
.
1.6.2.0
- Require
optparse-applicative-0.18.*
as the minimum and add an explicit dependency onprettyprinter
andprettyprinter-ansi-terminal
.
1.6.1.0
- Support building with
optparse-applicative-0.18.*
.
1.6.0.0
criterion-measurement-0.2.0.0
adds themeasPeakMbAllocated
field toMeasured
for reporting maximum megabytes allocated. Sincecriterion
re-exportsMeasured
fromCriterion.Types
, this change affectscriterion
as well. Naturally, this affects the behavior ofMeasured
’s{To,From}JSON
andBinary
instances.- Fix a bug in which the
--help
text for the--match
option was printed twice incriterion
applications.
1.5.13.0
- Allow building with
optparse-applicative-0.17.*
.
1.5.12.0
- Fix a bug introduced in version 1.5.9.0 in which benchmark names that include double quotes would produce broken HTML reports.
1.5.11.0
- Allow building with
aeson-2.0.0.0
.
1.5.10.0
- Fix a bug in which the
defaultMainWith
function would not use theregressions
values specified in theConfig
argument. This bug only affectedcriterion
the library—uses of the--regressions
flag fromcriterion
executables themselves were unaffected.
1.5.9.0
- Fix a bug where HTML reports failed to escape JSON properly.
1.5.8.0
-
The HTML reports have been reworked.
- The
flot
plotting library (js-flot
on Hackage) has been replaced byChart.js
(js-chart
). - Most practical changes focus on improving the functionality of the overview
chart:
- It now supports logarithmic scale (#213). The scale can be toggled by clicking the x-axis.
- Manual zooming has been replaced by clicking to focus a single bar.
- It now supports a variety of sort orders.
- The legend can now be toggled on/off and is hidden by default.
- Clicking the name of a group in the legend shows/hides all bars in that group.
- The regression line on the scatter plot shows confidence interval.
- Better support for mobile and print.
- JSON escaping has been made more robust by no longer directly injecting reports as JavaScript code.
- The
1.5.7.0
- Warn if an HTML report name contains newlines, and replace newlines with whitespace to avoid syntax errors in the report itself.
1.5.6.2
- Use unescaped HTML in the
json.tpl
template.
1.5.6.1
- Bundle
criterion-examples
’LICENSE
file.
1.5.6.0
- Allow building with
base-compat-batteries-0.11
.
1.5.5.0
- Fix the build on old GHCs with the
embed-data-files
flag. - Require
transformers-compat-0.6.4
or later.
1.5.4.0
-
Add
parserWith
, which allows creating acriterion
command-line interface using a customoptparse-applicative
Parser
. This is usefule for sitations where one wants to add additional command-line arguments to the default ones thatcriterion
provides.For an example of how to use
parserWith
, refer toexamples/ExtensibleCLI.hs
. -
Tweak the way the graph in the HTML overview zooms:
- Zooming all the way out resets to the default view (instead of continuing to zoom out towards empty space).
- Panning all the way to the right resets to the default view in which zero is left-aligned (instead of continuing to pan off the edge of the graph).
- Panning and zooming only affecs the x-axis, so all results remain in-frame.
1.5.3.0
- Make more functions (e.g.,
runMode
) able to print theµ
character on non-UTF-8 encodings.
1.5.2.0
-
Fix a bug in which HTML reports would render incorrectly when including benchmark names containing apostrophes.
-
Only incur a dependency on
fail
on old GHCs.
1.5.1.0
-
Add a
MonadFail Criterion
instance. -
Add some documentation in
Criterion.Main
aboutcriterion-measurement
’s newnfAppIO
andwhnfAppIO
functions, whichcriterion
reexports.
1.5.0.0
-
Move the measurement functionality of
criterion
into a standalone package,criterion-measurement
. In particular,cbits/
andCriterion.Measurement
are now incriterion-measurement
, along with the relevant definitions ofCriterion.Types
andCriterion.Types.Internal
(both of which are now under theCriterion.Measurement.*
namespace). Consequently,criterion
now depends oncriterion-measurement
.This will let other libraries (e.g. alternative statistical analysis front-ends) to import the measurement functionality alone as a lightweight dependency.
-
Fix a bug on macOS and Windows where using
runAndAnalyse
and other lower-level benchmarking functions would result in an infinite loop.
1.4.1.0
- Use
base-compat-batteries
.
1.4.0.0
-
We now do three samples for statistics:
performMinorGC
before the first sample, to ensure it’s up to date.- Take another sample after the action, without a garbage collection, so we can gather legitimate readings on GC-related statistics.
- Then
performMinorGC
and sample once more, so we can get up-to-date readings on other metrics.
The type of
applyGCStatistics
has changed accordingly. Before, it was:Maybe GCStatistics -- ^ Statistics gathered at the end of a run. -> Maybe GCStatistics -- ^ Statistics gathered at the beginning of a run. -> Measured -> Measured
Now, it is:
Maybe GCStatistics -- ^ Statistics gathered at the end of a run, post-GC. -> Maybe GCStatistics -- ^ Statistics gathered at the end of a run, pre-GC. -> Maybe GCStatistics -- ^ Statistics gathered at the beginning of a run. -> Measured -> Measured
When diffing
GCStatistics
inapplyGCStatistics
, we carefully choose whether to diff against the end stats pre- or post-GC. -
Use
performMinorGC
rather thanperformGC
to update garbage collection statistics. This improves the benchmark performance of fast functions on large objects. -
Fix a bug in the
ToJSON Measured
instance which duplicated the mutator CPU seconds where GC CPU seconds should go. -
Fix a bug in sample analysis which incorrectly accounted for overhead causing runtime errors and invalid results. Accordingly, the buggy
getOverhead
function has been removed. -
Fix a bug in
Measurement.measure
which inflated the reported time taken forperRun
benchmarks. -
Reduce overhead of
nf
,whnf
,nfIO
, andwhnfIO
by removing allocation from the central loops.
1.3.0.0
-
criterion
was previously reporting the following statistics incorrectly on GHC 8.2 and later:gcStatsBytesAllocated
gcStatsBytesCopied
gcStatsGcCpuSeconds
gcStatsGcWallSeconds
This has been fixed.
-
The type signature of
runBenchmarkable
has changed from:Benchmarkable -> Int64 -> (a -> a -> a) -> (IO () -> IO a) -> IO a
to:
Benchmarkable -> Int64 -> (a -> a -> a) -> (Int64 -> IO () -> IO a) -> IO a
The extra
Int64
argument represents how many iterations are being timed. -
Remove the deprecated
getGCStats
andapplyGCStats
functions (which have been replaced bygetGCStatistics
andapplyGCStatistics
). -
Remove the deprecated
forceGC
field ofConfig
, as well as the corresponding--no-gc
command-line option. -
The header in generated JSON output mistakenly used the string
"criterio"
. This has been corrected to"criterion"
.
1.2.6.0
-
Add error bars and zoomable navigation to generated HTML report graphs.
(Note that there have been reports that this feature can be somewhat unruly when using macOS and Firefox simultaneously. See https://github.com/flot/flot/issues/1554 for more details.)
-
Use a predetermined set of cycling colors for benchmark groups in HTML reports. This avoids a bug in earlier versions of
criterion
where benchmark group colors could be chosen that were almost completely white, which made them impossible to distinguish from the background.
1.2.5.0
- Add an
-fembed-data-files
flag. Enabling this option will embed thedata-files
fromcriterion.cabal
directly into the binary, producing a relocatable executable. (This has the downside of increasing the binary size significantly, so be warned.)
1.2.4.0
- Fix issue where
--help
would display duplicate options.
1.2.3.0
-
Add a
Semigroup
instance forOutliers
. -
Improve the error messages that are thrown when forcing nonexistent benchmark environments.
-
Explicitly mark
forceGC
as deprecated.forceGC
has not had any effect for several releases, and it will be removed in the next majorcriterion
release.
1.2.2.0
-
Important bugfix: versions 1.2.0.0 and 1.2.1.0 were incorrectly displaying the lower and upper bounds for measured values on HTML reports.
-
Have
criterion
emit warnings if suspicious things happen during mustache template substitution when creating HTML reports. This can be useful when using custom templates with the--template
flag.
1.2.1.0
-
Add
GCStatistics
,getGCStatistics
, andapplyGCStatistics
toCriterion.Measurement
. These are inteded to replaceGCStats
(which has been deprecated inbase
and will be removed in GHC 8.4), as well asgetGCStats
andapplyGCStats
, which have also been deprecated and will be removed in the next majorcriterion
release. -
Add new matchers for the
--match
flag:--match pattern
, which matches by searching for a given substring in benchmark paths.--match ipattern
, which is like--match pattern
but case-insensitive.
-
Export
Criterion.Main.Options.config
. -
Export
Criterion.toBenchmarkable
, which behaves like theBenchmarkable
constructor did prior tocriterion-1.2.0.0
.
1.2.0.0
-
Use
statistics-0.14
. -
Replace the
hastache
dependency withmicrostache
. -
Add support for per-run allocation/cleanup of the environment with
perRunEnv
andperRunEnvWithCleanup
, -
Add support for per-batch allocation/cleanup with
perBatchEnv
andperBatchEnvWithCleanup
. -
Add
envWithCleanup
, a variant ofenv
with cleanup support. -
Add the
criterion-report
executable, which creates reports from previously created JSON files.
1.1.4.0
-
Unicode output is now correctly printed on Windows.
-
Add Safe Haskell annotations.
-
Add
--json
option for writing reports in JSON rather than binary format. Also: various bugfixes related to this. -
Use the
js-jquery
andjs-flot
libraries to substitute in JavaScript code into the default HTML report template. -
Use the
code-page
library to ensure thatcriterion
prints out Unicode characters (like ², whichcriterion
uses in reports) in a UTF-8-compatible code page on Windows. -
Give an explicit implementation for
get
in theBinary Regression
instance. This should fix sporadiccriterion
failures with older versions ofbinary
. -
Use
tasty
instead oftest-framework
in the test suites. -
Restore support for 32-bit Intel CPUs.
-
Restore build compatibilty with GHC 7.4.
1.1.1.0
-
If a benchmark uses
Criterion.env
in a non-lazy way, and you try to use--list
to list benchmark names, you’ll now get an understandable error message instead of something cryptic. -
We now flush stdout and stderr after printing messages, so that output is printed promptly even when piped (e.g. into a pager).
-
A new function
runMode
allows custom benchmarking applications to run benchmarks with control over theMode
used. -
Added support for Linux on non-Intel CPUs.
-
This version supports GHC 8.
-
The
--only-run
option for benchmarks is renamed to--iters
.
1.1.0.0
-
The dependency on the either package has been dropped in favour of a dependency on transformers-compat. This greatly reduces the number of packages criterion depends on. This shouldn’t affect the user-visible API.
-
The documentation claimed that environments were created only when needed, but this wasn’t implemented. (gh-76)
-
The package now compiles with GHC 7.10.
-
On Windows with a non-Unicode code page, printing results used to cause a crash. (gh-55)
1.0.2.0
- Bump lower bound on optparse-applicative to 0.11 to handle yet more annoying API churn.
1.0.1.0
- Added a lower bound of 0.10 on the optparse-applicative dependency, as there were major API changes between 0.9 and 0.10.