keter
Web application deployment manager, focusing on Haskell web frameworks. It mitigates downtime.
LTS Haskell 22.39: | 2.1.5 |
Stackage Nightly 2024-10-29: | 2.1.8 |
Latest on Hackage: | 2.1.8 |
keter-2.1.8@sha256:2d9199c61b601313c9553b1e989b6c3818e9783c52258ee4eb71a781905fc833,4366
Module documentation for 2.1.8
- Keter
- Keter.App
- Keter.AppManager
- Keter.Cli
- Keter.Common
- Keter.Conduit
- Keter.Conduit.Process
- Keter.Config
- Keter.Context
- Keter.HostManager
- Keter.LabelMap
- Keter.Logger
- Keter.Main
- Keter.Plugin
- Keter.PortPool
- Keter.Proxy
- Keter.Rewrite
- Keter.TempTarball
- Keter.Yaml
Deployment system for web applications, originally intended for hosting Yesod applications. Keter does the following actions for your application:
- Binds to the main port (usually port 80) and reverse proxies requests to your application based on virtual hostnames.
- Provides SSL support if requested.
- Automatically launches applications, monitors processes, and relaunches any processes which die.
- Provides graceful redeployment support, by launching a second copy of your application, performing a health check[1], and then switching reverse proxying to the new process.
Keter provides many more advanced features and extension points. It allows configuration of static hosts, redirect rules, management of PostgreSQL databases, and more. It supports a simple bundle format for applications which allows for easy management of your web apps.
[1]: The health check happens trough checking if a port is opened. If your app doesn’t open a port after 30 seconds it’s presumed not healthy and gets a term signal.
Quick Start
To get Keter up-and-running quickly for development purposes, on an Ubuntu system (not on your production server), run:
wget -O - https://raw.githubusercontent.com/snoyberg/keter/master/setup-keter.sh | bash
(Note: This assumes you already have keter installed via cabal.)
(Note: you may need to run the above command twice, if the shell exits after
apt-get
but before running the rest of its instructions.) This will download
and build Keter from source and get it running with a
default configuration. By default Keter will be set up to support HTTPS and
will require you to provide a key and certificate in /opt/keter/etc
. You can
disable HTTPS in /opt/keter/etc/keter-config.yaml
by commenting the certificate
and key lines.
This approach is not recommended for a production system. We do not recommend
installing a full GHC toolchain on a production server, nor running such ad-hoc
scripts. This is intended to provide a quick way to play with Keter, especially
for temporary virtual machines. For a production system, we recommend building
the keter
binary on a separate system, and tracking it via a package manager
or similar strategy.
Bundling your app for Keter
-
Modify your web app to check for the
PORT
environment variable, and have it listen for incoming HTTP requests on that port. Keter automatically assigns arbitrary ports to each web app it manages. When building an app based on the Yesod Scaffold, it may be necessary to change theport
variable inconfig/settings.yaml
fromYESOD_PORT
toPORT
for compatibility with Keter. -
Create a file
config/keter.yaml
. The minimal file just has two settings:exec: ../path/to/executable host: mydomainname.example.com
See the bundles section below for more available settings.
-
Create a gzipped tarball with the
config/keter.yaml
file, your executable, and any other static resources you would like available to your application. This file should be given a.keter
file extension, e.g.myapp.keter
. -
Copy the
.keter
file to/opt/keter/incoming
. Keter will monitor this directory for file updates, and automatically redeploy new versions of your bundle.
Examples are available in the incoming directory.
Setup
Building keter for Debian, Ubuntu and derivatives
Eventually, I hope to provide a PPA for this (please contact me if you would like to assist with this). For now, the following steps should be sufficient:
First, install PostgreSQL:
sudo apt-get install postgresql
Second, build the keter
binary and place it at /opt/keter/bin
. To do so,
you’ll need to install the Haskell Platform, and can then build with cabal
.
This would look something like:
sudo apt-get install haskell-platform
cabal update
cabal install keter
sudo mkdir -p /opt/keter/bin
sudo cp ~/.cabal/bin/keter /opt/keter/bin
Third, create a Keter config file. You can view a sample at https://github.com/snoyberg/keter/blob/master/etc/keter-config.yaml.
Optionally, you may wish to change the owner on the /opt/keter/incoming
folder to your user account, so that you can deploy without sudo
ing.
sudo mkdir -p /opt/keter/incoming
sudo chown $USER /opt/keter/incoming
Building keter for Redhat and derivatives (Centos, Fedora, etc)
First, install PostgreSQL:
sudo dnf install postgresql
Second, build the keter
binary and place it at /opt/keter/bin
. To do so,
you’ll need to install the Haskell Platform, and can then build with cabal
.
This would look something like:
sudo dnf install haskell-platform
cabal update
cabal install keter
sudo mkdir -p /opt/keter/bin
sudo cp ~/.cabal/bin/keter /opt/keter/bin
Third, create a Keter config file. You can view a sample at https://github.com/snoyberg/keter/blob/master/etc/keter-config.yaml.
Configuring startup
For versions of Ubuntu and derivatives 15.04 or greater and Redhat and derivatives (Centos, Fedora, etc) use systemd
# /etc/systemd/system/keter.service
[Unit]
Description=Keter
After=network.service
[Service]
Type=simple
ExecStart=/opt/keter/bin/keter /opt/keter/etc/keter-config.yaml
[Install]
WantedBy=multi-user.target
Finally, enable and start the unit (Note: You may need to disable SELinux):
sudo systemctl enable keter
sudo systemctl start keter
Verify that it’s actually running with:
sudo systemctl status keter
Optionally, you may wish to change the owner on the /opt/keter/incoming
folder to your user account, so that you can deploy without sudo
ing.
sudo mkdir -p /opt/keter/incoming
sudo chown $USER /opt/keter/incoming
Additionally, you may want to enable logging to stderr by disabling rotate-logs
in config/keter.yaml
, since systemd will automatically capture and manage stderr output for you:
rotate-logs: false
For versions of Ubuntu and derivatives less than 15.04, configure an Upstart job.
# /etc/init/keter.conf
start on (net-device-up and local-filesystems and runlevel [2345])
stop on runlevel [016]
respawn
# NB: keter writes logs to /opt/keter/log, but some exceptions occasionally
# escape to standard error. This ensures they show up in system logs.
console output
exec /opt/keter/bin/keter /opt/keter/etc/keter-config.yaml
Finally, start the job for the first time:
sudo start keter
NixOS
Keter is integrated within nixos:
There is an example that integrates yesod into keter with NixOS here: https://github.com/jappeace/yesod-keter-nix
Bundles
An application needs to be set up as a keter bundle. This is a GZIPed tarball
with a .keter
filename extension and which has one special file:
config/keter.yaml
. A sample file is available at
https://github.com/snoyberg/keter/blob/master/incoming/foo1_0/config/keter.yaml.
Keter also supports wildcard subdomains and exceptions, as in this example configuration:
exec: ../com.example.app
args:
- Hello
- World
- 1
host: www.example.com
extra-hosts:
- "*.example.com"
- foo.bar.example.com
static-hosts:
- host: static.example.com
root: ../static
redirects:
- from: example.com
to: www.example.com
Due to YAML parsing, wildcard hostnames will need to be quoted as above.
Wildcard hostnames are not recursive, so foo.bar.example.com
must be
explicitly added as an extra hostname in the above example, or
alternatively, *.*.example.com
would cover all host names two levels
deep. It would not cover host names only one level deep, such as
qux.example.com
. In this manner, wildcard hostnames correspond to the
manner in which SSL certificates are handled per RFC2818. Wildcards may
be used in only one level of a hostname, as in foo.*.example.com
.
Full RFC2818 compliance is not present - f*.example.com
will not be
handled as a wildcard with a prefix.
A sample Bash script for producing a Keter bundle is:
#!/bin/bash -ex
cabal build
strip dist/build/yesodweb/yesodweb
rm -rf static/tmp
tar czfv yesodweb.keter dist/build/yesodweb/yesodweb config static
For users of Yesod, The yesod
executable provides a keter
command for
creating the bundle, and the scaffolded site provides a keter.yaml
file.
Deploying
In order to deploy, you simply copy the keter bundle to /opt/keter/incoming
.
To update an app, copy in the new version. The old process will only be
terminated after the new process has started answering requests. To stop an
application, delete the file from incoming.
PostgreSQL support
Keter ships by default with a PostgreSQL plugin, which will handle management of PostgreSQL databases for your application. To use this, make the following changes:
- Add the following lines to your
config/keter.yaml
file:
plugins:
postgres: true
- Keter can be configured to connect to a remote postgres server using the following syntax:
plugins:
postgres:
- server: remoteServerNameOrIP
port: 1234
Different webapps can be configured to use different servers using the above syntax. It should be noted that keter will prioritize it’s own postgres.yaml record for an app. So if moving an existing app from a local postgres server to a remote one (or switching remote servers), the postgres.yaml file will need to be updated manually.
Keter will connect to the remote servers using the postgres
account. This setup
assumes the remote server’s pg_hba.conf
file has been configured to allow connections
from the keter-server IP using the trust
method.
(Note: The plugins
configuration option was added in v1.0 of the
keter configuration syntax. If you are using v0.4 then use postgres: true
.
The remote-postgres server syntax was added in v1.4.2.)
-
Modify your application to get its database connection settings from the following environment variables:
PGHOST
PGPORT
PGUSER
PGPASS
PGDATABASE
-
The Yesod scaffold site is already equipped to read these environment variables when they are set.
Known issues
-
There are reports of Keter not working behind an nginx reverse proxy. From the reports, this appears to be a limitation in nginx’s implementation, not a problem with Keter. Keter works fine behind other reverse proxies, including Apache and Amazon ELB.
One possible workaround is to add the following lines to your nginx configuration:
proxy_set_header Connection ""; proxy_http_version 1.1;
This has not yet been confirmed to work in production. If you use this, please report either its success or failure back to me.
Additionally, to make sure that nginx does not reset the
Host
header (which keter uses to choose the right target), you will need to add:proxy_set_header Host $host;
-
Keter does not handle password-protected SSL key files well. When provided with such a key file, unlike Apache and Nginx, Keter will not pause to ask for the password. Instead, your https connections will merely stall.
To get around this, you need to create a copy of the key without password and deploy this new key:
openssl rsa -in original.key -out new.key
(Back up the original key first, just in case.)
Stanza-based config files
Starting with Keter 1.0, there is an alternate format for application Keter config files, which allows much more flexibility in defining multiple functionality for a single bundle (e.g., more than one web app, multiple redirects, etc). This README will eventually be updated to reflect all various options. In the meanwhile, please see the following examples of how to use this file format:
- https://github.com/yesodweb/yesod-scaffold/blob/postgres/config/keter.yml
- https://github.com/snoyberg/keter/blob/master/incoming/foo1_0/config/keter.yaml
Multiple SSL Certificates
Keter is able to serve different certificates for different hosts,
allowing for the deployment of distinct domains using the same
server. An example keter-config.yaml
would look like::
root: ..
listeners:
- host: "*4" # Listen on all IPv4 hosts
port: 80
- host: 127.0.0.1
key: key.pem
certificate: certificate1.pem
- host: 127.0.0.2
key: key.pem
certificate: certificate2.pem
An alternative way to make this possible is adding the following ssl:
argument
to the keter.yaml
file in your Yesod app’s config folder
as follows:
stanzas:
- type: webapp
exec: ../yourproject
ssl:
key: /opt/keter/etc/cert/yourproject.key
certificate: /opt/keter/etc/cert/yourproject.crt
chain-certificates: []
If you don’t have your certificates bundled in one .crt
file, you should add
the other certificates in the following order
ssl:
[..]
chain-certificates:
- /opt/keter/etc/middle.crt
- /opt/keter/etc/root.crt
This way you can designate certificates per Yesod App while still having one SSL certificate
in your main /opt/keter/etc/keter-config.yaml
for your other Yesod apps to default to
if they don’t have this ssl:
argument in their config/keter.yaml
.
NOTE: If you get an error that a Bool was expected instead of an Object when adding the ssl:
argument, then for this to work you might need to build Keter from Github, because at the time
of writing the version of Keter on Hackage does not have this functionality. Just clone or
download this repository and build it using stack.
FAQ
- Keter spawns multiple failing process when run with
sudo start keter
.- This may be due to Keter being unable to find the SSL certificate and key.
Try to run
sudo /opt/keter/bin/keter /opt/keter/etc/keter-config.yaml
. If it fails withketer: etc/certificate.pem: openBinaryFile: does not exist
or something like it, you may need to provide valid SSL certificates and keys or disable HTTPS, by commenting the key and certificate lines from/opt/keter/etc/keter-config.yaml
.
- This may be due to Keter being unable to find the SSL certificate and key.
Try to run
Debugging
There is a debug port option available in the global keter config:
cli-port = 1234
This allows you to attach netcat to that port, and introspect which processes are running within keter:
nc localhost 1234
Then type --help
for options, currently it can only list
the apps, but this approach is easily extensible
if you need additional debug information.
This option is disabled by default, but can be useful to figure out what keter is doing.
Contributing
If you are interested in contributing, see https://github.com/snoyberg/keter/blob/master/incoming/README.md for a complete testing workflow. If you have any questions, you can open an issue in the issue tracker, ask on the #yesod freenode irc channel, or send an email to [email protected].
Changes
2.1.8
- Bump tls, make it build with new release
2.1.7
- add lower limit for tar to 0.6.0.0, which fixes “FileNotExecutable” bug. see #291, special thanks to @scheottl for finding and debugging.
2.1.6
- Bump network
2.1.5
- Fix OOM bug on logfile rotation
2.1.4
- bump package versions, tls, warp and zlib
2.1.3
-
Add
healthcheck-path
global config for Keter-only healthchecks. PR #283 -
Fix config keys
unknown-host-response-file
andmissing-host-response-file
accidentally flipped. PR #282 -
In case reading any one of
*-host-response-file
fails, keter now logs a warning, and falls back to builtin defaults. Before 2.1.3, this is a fatal error. -
Add support for tar 0.6, drop NIH tar unpack.
- Change CI to be cabal based instead of stack.
2.1.2
- Bump bounds:
aeson >=2.0.0 && <2.2 (latest: 2.2.0.0) bytestring >=0.10.12 && <0.12 (latest: 0.12.0.0) mtl >=2.2.2 && <2.3 (latest: 2.3.1) optparse-applicative >=0.16.1 && <0.18 (latest: 0.18.1.0) transformers >=0.5.6 && <0.6 (latest: 0.6.1.1) unix >=2.7.2 && <2.8 (latest: 2.8.1.1) warp-tls >=3.0.3 && <3.4.0 (latest: 3.4.0)
2.1.1
- Bump unix-compat bound to accept 0.7
- Add upper bounds to all dependencies in the main library
- Run cabal-fmt
- Drop support for stackage lts-17 and lts-18 I don’t think it’s worth mainting these with the aeson issues involved. Eg you open yourself to a DoS attack by staying on these old versions
2.1
Please reference MigrationGuide-2.1.md
for in-depth documentation on breaking changes to be aware of, examples of said changes, and potential solutions/workarounds for them if you plan on upgrading to this version of keter
.
- Log naming and directory scheme has changed for both main keter logs and app logs.
Old logs were nameddir/current.log
for the current log and%Y%m%d_%H%M%S.log
(time
package conventions) for rotated logs.
Current logs have been brought up one level and named after their old directory:
logs/keter/current.log
->logs/keter.log
Rotated logs will now simply have.1
.2
ascending appended to the name of the base logs rather than be named after the date and time they were rotated at:
logs/keter/20230413_231415.log
->logs/keter.log.1
logs/__builtin__/20230413_231415.log
->logs/__builtin__.log.1
logs/app-foo/20230413_231415.log
->logs/app-foo.log.1
Please update anything that depended on the old log naming and directory conventions accordingly. - Added the
rotate-logs
option (default: true) in the keter config file.
When true, the main keter (non-app!) logs will rotate like they have in previous versions.
When false, the main keter logs will emit straight to stderr; this is useful e.g. if you’re running keter via systemd, which captures stderr output for you. - Internal logging implementation has been switched over to
fast-logger
instead of the old in-house logging solution.
Please be aware that compared to the old logging solution, the usage offast-logger
does not 100% guarantee consistent time ordering. If anything previously depended on tailing the last line of a log and critically assumed that messages will be in order, it should now parse via timestamp instead. - The
LogMessage
ADT has been removed. - Replaced individual logging calls with TH splice -style calls where sensible to have access to source location info.
- Updated log message format to make use of the additional info:
"$time|$module$:$line_num|$log_level> $msg"
- Added
Keter.Context
, exposing the newKeterM
monad and related functions.
This monad carries a mappable global config and logger around, eliminating the need to pass various configuration data and the logger to everything. - Refactored most
Keter.*
module functions to be actions inKeterM
Please anticipate breaking changes for anything written against the exposed API.
2.0.1
- Force usage of http-reverse-proxy versions above 0.6.0.1. This prevents a DoS attack on a head request followed by a post.
2.0
- Improve missing sudo error messages in postgres plugin.
- Reorganized most Haskell source files into /src.
- Dropped support for http-client < 0.5.0.
- Removed ‘default’ package.
- All “Data” modules are now “Keter” modules.
- Testing library switched from “hspec” to “tasty”.
- Move Network.Http.ReverseProxy.Rewrite into Keter.Rewrite
- Move Codec.Archive.TempTarball into Keter.TempTarball
- Hide Keter.Aeson.KeyHelper
- Stop re-exporting common and rewrite from types
- Common no longer re-exports half of Haskell
- Rename Types to Config
- Move Common out of Config into root
1.9
- Update status code of missing host responses. They now emit a 502 on missing host, and 404 on host not found
- Always restart keter in the nix config for systemd. It turns out that keter may exit with exit code 0 under load testing. Changing from on-failure to always in systemd should bring it back up.
- Squash proxy exceptions if they occur and serve a default or custom error response. Emits the exception to the log.
- Add incoming folder to the CI build.
1.8.4
- Get rid of ominious warning at the top. Thanks to /u/Opposite-Platypus-99 for pointing this out.
1.8.3
- HTML escape X-forwarded-host response as well.
1.8.2
- Fix XSS issue in the default response for host not found. (special thanks to Max @ulidtko for spotting and fixing this)
1.8.1
- Fix haddock build
1.8
- Add NixOS support
- Describe debug port in readme.
- Improve ensure alive error message due to https://github.com/snoyberg/keter/issues/236
- Add
missing-host-response-file
andunknown-host-response-file
to the global keter config, which replace the default responses. - All missing-host responses will now fill the requested host in the
X-Forwarded-Host: HOSTNAME
header, where HOSTNAME is the requested host. This is done because the default response fills in the hostname. Now javascript could potentially fix that by making another request to itself. - Document missing configuration options in
etc/keter-config.yaml
1.7
- Add support Aeson 2.*
- Add
Data.Aeson.KeyHelper.hs
in cabal file. - And use the module where Aeson has changed how to handle Key and KeyMap.
1.6
- Make keter more chatty on boot. This allows you to figure out in code where things go wrong.
- Add opt-in debug CLI, allowing you to inspect keters’ internal state. You can activate it by specifying a cli-port.
- Emit which pid is being killed by keter.
This helps with process leakage issues,
for example if the user launches from a bash script without using
exec
.
1.5
- Builds with
process
1.6 - add dependency for
tls-session-manager
- Add show instance for App
- Add ensure alive timeout config
- Add
nc
example in incoming - Change to github actions because travis ci stopped working.
- Fix hackage issues in cabal file
- Fix breaking changes with warp-tls.
1.4.3.1
- Add cabal flag
system-filepath
for compatibility with older versions of fsnotify.
1.4.3
- Update fsnotify dependency version and remove system-filepath.
1.4.2.1
Bug fix: Change default connection time bound from 5 sec to 5 minutes #107
1.4.1
- Add configurable timeouts #93
1.4.0.1
- Avoid infinite loop traversing incoming directory #96
1.4.0
- Drop system-filepath
1.3.10
- Configurable time bound #92
1.3.9.2
- Lower case PostgreSQL names #88
1.3.9.1
- Allow blaze-builder 0.4
1.3.9
- Support chain certificates in credentials #82
1.3.7.1
Bug fix: catch exceptions during reload #64
1.3.7
- Add ability to use middleware #63
1.3.6
Support the forward-env
setting.
1.3.5.3
More correct/complete solution for issue #44. Allows looking up hosts either with or without port numbers.
1.3.5.2
Partial workaround for keter.yaml files that give a port with the hostname.
1.3.5.1
Fix bug where the cleanup process would remain running.
1.3.5
All stanzas may have the requires-secure
property to force redirect to HTTPS. You can set additional environment variables in your global Keter config file.
1.3.4
Support for overriding external ports. Support for keter.yml in addition to keter.yaml. Case insensitive hostname lookups.
1.3.3
Set the X-Forwarded-Proto header
1.3.2
Enable GZIP middleware
1.3.1
Upgrade to WAI 3.0
1.3.0
Upgrade to conduit 1.1
1.0.1
Permit use of wildcard subdomains and exceptions to wildcards. Convert internal strings to use Data.Text in more places. (Although internationalized domain names are not supported unless entered in punycode in configuration files.)
1.0.0
Significant overhaul. We now support monitoring of much more arbitrary jobs (e.g., background tasks), have a proper plugin system (PostgreSQL is no longer a required component), and have a much better system for tracking hostname mapping changes.
0.4.0
Switch to fsnotify to get cross-platform support. No longer using raw proxies, but instead WAI proxies.
0.3.7
Sending a HUP signal reloads the list of deployed apps. This is useful for circumstances where inotify does not work correctly, such as on file systems which do not support it.
0.3.5
You can now create Keter bundles without any applications. These can contain static hosts and redirects.