About
This is a library that facilitates sending email via AWS SES using the
background processor hworker. In
particular, it handles rate limiting (for sending rate currently, not
daily quotas), as SES does not queue messages.
Rate limiting
Aside from sending emails in the background, the main thing that
hworker-ses provides is rate limiting. The rate limiting is only on
a per-process (most likely, per server) basis, as it is controlled via
in-memory storage. This means that you should take into account how
many workers you are likely to have when setting the
messages-per-second rate on the workers.
For example, if your rate limit is 30 messages per second, and you expect
to have at most 5 workers (you can organize them however you like, but a
simple strategy is just to start a worker with each application process),
then if you set the rate limit to be 6 messages per second, things will be
fine.
Note that if you do run over, it won’t mean that your messages actually
don’t get delivered. That error code will trigger a retry on the message,
so eventually it will go out. But it’s wasteful (as you can know in advance
if a given message will actually be able to be sent), and if you really
mess it up (ie, set an absurdly high limit), you potentially will have
your API calls rate limited, which isn’t good.
Usage
The program in the example directory is probably most of what you
need to get started. In particular, you create an hworker (you need
to give it a name for the queue - this should be shared by any servers
that should be accessing the same queue and sending the same messages,
though if you are sharing the redis server with other applications,
this should be distinct). The third argument is the per-second rate
limit for this server, and the last argument is the address that the
messages are sent from.
Using what you get back, you can spawn workers and a monitor. Then you
can queue as many messages as you want, and they will be sent out by
the workers.
You need to start at least one worker thread and at least
one monitor thread. You can have as many of these as you want, though
generally speaking, having more than one monitor thread per server is
probably overkill (for the system to work, you want at least one
monitor running somewhere - so if a whole server crashes, you want
there to be a monitor elsewhere). Having many workers makes things run
faster. In particular, if you have a high sending limit, you will
probably want several workers.
As specified in the hworker
documentation, the semantics of this queue is at-least-once, so it’s
possible that messages can get sent multiple times in error conditions
(like if an entire server crashes right after the message is sent, but
before the fact that it was sent is acknowledged). But, provided that
redis is still available and is reliable, messages that have been
queued are guaranteed to get sent eventually (even in the case of
servers crashing, etc). The only caveat to that is that for the
message to get sent at least one worker and one monitor thread must be
running (thoses don’t need to be running all the time, but as long as
they aren’t running, messages might be delayed. Once they’re started again,
those delayed messages will go out).
Primary Libraries Used
Contributors
