goldplate
A lightweight golden test runner
https://github.com/jaspervdj/goldplate
Version on this page: | 0.2.1.1@rev:5 |
LTS Haskell 22.33: | 0.2.1.1@rev:6 |
Stackage Nightly 2024-09-07: | 0.2.1.1@rev:6 |
Latest on Hackage: | 0.2.1.1@rev:6 |
goldplate-0.2.1.1@sha256:3640ebec677888158370f748ab8b7602b4d7426715107950502cafc147a4a5d9,2149
Module documentation for 0.2.1.1
There are no documented modules for this package.
goldplate 🏅
$ goldplate -j2 tests/
1..26
ok tests/prettify-json.goldplate: exit_code
ok tests/prettify-json.goldplate: stdout
ok tests/env.goldplate: exit_code
ok tests/env.goldplate: stdout
...
goldplate
is a cute and simple opaque golden test runner for CLI
applications. You place your test cases in a directory, annotate them with
.goldplate
files, and that’s it. It is completely language agnostic. And
perhaps its best feature is that it can automatically✨🪄 fix your tests outputs!
Give goldplate
a try if:
- You are testing a CLI application
- The application produces deterministic output (e.g. compilers, prettifiers, convertors)
- The application is quick to start (a process is created for every test)
- Your output is text-based and not huge in size
At Fugue, we’ve been using internal versions of this tool since 2016, so it should be pretty stable.
goldplate
produces output compatible with the Test Anything Protocol.
Table of Contents
Tutorial
Creating a first test
Imagine we are up to testing the behaviour of echo
command. In this very
simple example, we run echo "Hello, world!"
and expect it to print Hello, world!
to the stdout stream as a result.
Create a new file echo.goldplate
and add the following content:
{
"command": "echo",
"arguments": ["Hello, world!"],
"asserts": [
{"exit_code": 0},
{"stdout": "hello-world.txt"}
]
}
Let’s go through this bit by bit.
The test invocation is specified by the command
and arguments
fields. We
are invoking the echo
process with a single argument, "Hello, world!"
.
The expected results of our test live in the asserts
field. This simple test
has two asserts:
- We verify that the exit code is 0 (success).
- We check the
stdout
(output) of the command against the contents of the filehello-world.txt
.
We haven’t created hello-world.txt
yet, but that’s not a problem. We can
invoke goldplate --fix
to create it:
$ goldplate echo.goldplate --pretty-diff --fix
1..2
ok echo.goldplate: exit_code
not ok echo.goldplate: stdout
diff:
0a1
> Hello, world!
fixed ./hello-world.txt
After hello-world.txt
has been created with proper contents, subsequent
goldplate
invocations will pass:
$ goldplate echo.goldplate
1..2
ok echo.goldplate: exit_code
ok echo.goldplate: stdout
You can view the full example here:
Feeding input on stdin
View example:
You can pass one or multiple lines of input to the command by using the stdin
field.
Setting environment variables
View example:
The environment
field can be used to set environment variables for the
program.
goldplate
also sets a number of environment
variables. You can use these directly within the
configuration JSON. In this example, we use:
{"stdout": "${GOLDPLATE_NAME}.stdout"}
Rather than:
{"stdout": "env.stdout"}
We found this to be good practice, it makes mass-renaming of tests much easier.
Setting work directory
View example:
The working_directory
field can be used to set the work directory in which the
command will be executed. It can either be an absolute path or a path relative
to the goldplate
file. If a working_directory
is specified then the other
fields like input_files
and stdout
need to be relative to the
working_directory
as well.
If a work directory is not specified the command
will be executed in the
same directory as the goldplate
file.
Globbing input files
View example:
.goldplate
files are fairly small but if you have a whole directory of files
that you just want to run the same command on, they can get very repetitive.
This is why goldplate
provides a simple way to pull in multiple input files.
If the input_files
field is set to a glob, all asserts will be ran for every
matching input file. goldplate
will set the following variables:
${GOLDPLATE_INPUT_FILE}
: the path to the input file${GOLDPLATE_INPUT_NAME}
: the input file without extension
Post processing: find and replace
View example:
Sometimes you may want to do a find-and-replace on the actual output, for example to filter out timestamps or other information that you do not expect to match up against the expected output.
Post processing: prettify JSON
View example:
Many modern CLI tools output JSON. You can use the prettify_json
post
processor to make sure the JSON is converted to a normalized form with sorted
keys and consistent indentation.
Created files and directories
View example:
goldplate
is geared towards checking the stdout
and stderr
outputs of a
program, but you can also check that files were created with specific contents.
If you do this, goldplate
will remove these files and directories afterwards
to leave a clean repository behind.
Installation
Binaries for Mac OS and Linux are available on the releases page.
Installation from source is done using standard Haskell tooling – Cabal and stack both work well.
Using stack
- Install stack for your platform.
- Clone this repository and
cd
into it. - Run
stack install
. - Make sure
$HOME/.local/bin
is in your$PATH
.
Using Cabal
- Install Cabal for your platform.
- Clone this repository and
cd
into it. - Run
cabal install
. - Make sure
$HOME/.cabal/bin
is in your$PATH
.
Reference
Syntax
Environment variables can be spliced into the configuration using ${VAR}
syntax within strings. To escape this syntax, use $${VAR}
to get a literal
${VAR}
, $$${VAR}
to get a literal $${VAR}
, and so on.
Environment variables
The test is always executed in the directory that holds the .goldplate
file.
goldplate
will always set the following environment variables:
GOLDPLATE_FILE
: The filename of the.goldplate
file.GOLDPLATE_NAME
: The filename of the.goldplate
file without the extension.GOLDPLATE_BASENAME
: The basename (filename without directory) of the.goldplate
file.
When dealing with multiple input files, the following additional variables are set:
GOLDPLATE_INPUT_FILE
: The input file name (relative to the current directory).GOLDPLATE_INPUT_NAME
: The same asGOLDPLATE_INPUT_FILE
but without the extension.GOLDPLATE_INPUT_BASENAME
: The basename (filename without directory) of the input file.
Here is an example that outputs all of these environment variables:
Similar projects
A similar project is smoke. I think goldplate
has two major advantages
over smoke:
- It can fix the output files automatically using
--fix
! This is very useful if you make a change to your tool that affects a lot of test files. You still need to manually review the diff, but at least you don’t need to manually update the specs. - You can avoid most repetitive configs by using simple globbing.
Changes
CHANGELOG
-
0.2.1.1 (2023-02-26)
- Bump
aeson
dependency upper bound to 2.1. - Tested with GHC 8.4 - 9.6.1 alpha3.
- Bump
-
0.2.1 (2022-04-12, github release only)
- Print summary line at the end of the TAP format.
- Add
--version
option (by Andreas Abel). - Include testsuite in package (by Andreas Abel).
- Tested with GHC 8.4 - 9.2.
-
0.2.0 (2021-05-24)
- Change output to Test Anything Protocol.
- Add
working_directory
field (by Beatrice Vergani).
-
0.1.3 (2021-02-10)
- Bump
aeson
dependency upper bound to 1.5. - Bump GHC to 8.10.3.
- Bump
-
0.1.2 (2020-11-05)
- Bump
regex-pcre-builtin
dependency lower bound to 0.95.1.3 to fix issue withText
regexes (by Stefano Debenedetti). - Fix issue with
replace
in post process step. If the string was not matched at all, or if there was a remainder, it could get dropped (by Stefano Debenedetti).
- Bump
-
0.1.1 (2020-06-29)
- Add
GOLDPLATE_BASENAME
andGOLDPLATE_INPUT_BASENAME
environment variables. - Fix issue with
GOLDPLATE_FILE
environment variable.
- Add
-
0.1.0 (2020-06-21)
- Initial release.