katip-elasticsearch is a scribe for the Katip logging framework that
sends structured logs to ElasticSearch.
Features
Built in bounded buffering.
Configurable pool of logging workers to help with high write
volume.
Optional field type annotation to avoid mistyping values.
Optional automatic date sharding, so logs can be filed into monthly,
weekly, daily, hourly, every minute indices. You can even specify
your own index routing logic. This pattern can be seen in the ELK
stack as a way of keeping indexes reasonably sized and easy to
optimize, rotate, and manage.
Customizable retry policy for temporary outages and errors.
Automatic index and mapping setup.
Changes
0.3.0.1
Loosen deps on bloodhound, aeson, and http-client.
0.3.0.0
Added zero padding to date-based indices. This shouldn’t negatively
impact most users but to be safe, this was put behind a breaking
version number. Previously, you may see indices
log-index-prefix-2016-1-2. That index will now be
log-index-prefix-2016-01-02, so at worst when you deploy, the day
of the change will have 2 indices: one zero-padded and one not. If
you are using custom index sharding, you will not be affected by
this. The reasoning here is that most existing elasticsearch tools
such as curator expects date indices to be zero padded. By
switching to the standard, users can easily get log rotation and
other features from curator rather than having to roll their own
or add them to Katip.
0.2.1.0
Drop direct dependency on random, upgrade uuid to >= 1.3.12 for
safer id generation. Previously, UUID was using randomIO, which uses
the system clock as a seed. So if multiple nodes happened to start
at the same time, they would produce conflicting UUID sequences.
0.2.0.0
Default index sharding policy to daily. Previously it was no
sharding. The reasoning here is that no sharding creates very large
indices which become very difficult to manage in
production. Rotating data out on a time basis is very slow compared
to deleting date-based indices.
Upgrade note: if you were using the defaults before and switch to
daily, rather than having the index name of my-index, you’ll start
seeing my-index-2016-3-14. The good news is that whatever you’re
using to use to search against your logs (such as kibana) will
support index patterns, so just use the pattern of my-index* to
get everything. Eventually if you have a retention period, you can
manually delete the my-index index without disruption.
