New Stackage snapshot format

I’ve been blogging off and on about Pantry and how it affects both Stack and Stackage. There have been a number of Github issues lying around about changes to how we’ll build Stackage snapshots. Most of those changes have landed, and it’s worth recapping the problems we were trying to solve, how things used to work, how they work now, and what’s left.

Old build process overview

The stackage repo has a build-constraints.yaml file, which specifies which packages should be included in the next nightly, which tests are expected to fail, upper bounds, etc. The file is where most people interact with the Stackage project, where pull requests land, and where the Stackage Curator team applies changes.

The stackage-curator tool is used on Travis CI to put together the constraints file with the current state of Hackage to come up with a snapshot, and then check to make sure all of the bounds on that snapshot are valid. When you get an issue from someone on the Curator team reporting bounds failures, it’s a report from stackage-curator.

We run a build server that rebuilds nightly snapshots on a regular basis. It uses the stackage-curator tool as well to create the snapshot definition, perform the build, run tests, upload Haddocks to Amazon S3, and commit the new snapshot file to the stackage-nightly repo. If there are build failures, the Stackage Curator team reports on them and adds more constraints to work around them.

LTS snapshots are basically the same thing, except for an LTS minor version bump, the constraints come from the previous snapshot (e.g., lts-13.22 is based on the constraints implicit in lts-13.21), and the snapshots are committed to the lts-haskell repo.

Finally, Stackage Server (a.k.a. stackage.org) takes information from stackage-nightly, lts-haskell, and the uploaded docs on S3 to provide everything it does: lists of snapshots, packages, Haddocks, and Hoogle search.

Problems

stackage-curator implements its own Cabal package building logic by working directly with Cabal-the-library. There’s a lot of feature overlap with Stack, and different behavior in some corner cases.
The original Stackage file format used in stackage-nightly and lts-haskell is unique from the “custom snapshot” format that Stack supports, for only historical reasons.
The snapshot format is big. It contains a lot of information about module names, constraints, users, etc, that build tools don’t need.
While there is cryptographic hash information on cabal files to deal with Hackage Revisions, there is no cryptographic hash information on the package tarball contents themselves.

Changes

We’re scrapping the previous stackage-curator tool entirely.
There’s a brand new curator tool, which uses the same snapshot format as Stack 2, Pantry, and what was previously known as custom snapshots.
This curator tool still does bounds checking, and is now included in Travis CI. However, this tool does not build packages. It shells out to Stack to do that instead. That makes this tool much simpler.
Important end user consideration: the old tool would automatically add dependencies of packages to a snapshot. After some discussion and lots of requests around this, all packages to be included in a snapshot must appear explicitly in build-constraints.yaml. Some people have already gotten reports about this, which is what initially spurred this blog post.
New snapshots are landing in the stackage-snapshots repo.
For backwards compatibility, we currently convert snapshots to the original format in the original repos, so stackage-nightly and lts-haskell are still being populated. This will not continue indefinitely, we do not have a timeframe on when we will discontinue this.

Current status

We’ve moved over Nightly builds completely to the new tool. We have not yet moved LTS Haskell over. Due to differences in how constraints are tracked, we’re going to make the switchover when we release LTS 14. We’ll put out a separate post about LTS 14 fairly soon.

Takeaways

The two most important takeaways for most people are: