bytezap

Build strict bytestrings with zero intermediate allocation.

If you’re looking for general high-performance serialization, you probably want mason. But if you’re dealing with data that is already “shaped” like binary data, e.g. using types defined in binrep, and you want the best performance possible, read on…

Why?

Most binary serialization libraries tend towards a model where the serializer itself handles allocation. In the plumbing, serialization operations are bracketed by a check to ensure the current buffer has enough space for the next operation; if not, we obtain more space in some way, and serialization continues. This design is nice because we can chunk the serializing:

• for writing to a lazy bytestring, we can emit a new chunk and clear our buffer
• for writing to a handle, we can write, flush and clear our buffer
• for writing to a strict bytestring, we must grow our current buffer (meh)

But if we know the size of the serialized data before serializing it, we don’t need those space checks, nor these intermediate steps. We may allocate a single buffer with the required size upfront, then use that as we like.

Great, you say, but most data isn’t so simple that we can easily calculate its serialized length without actually performing the serialization. This is true. bytezap is designed specifically for cases where

• it’s easy to calculate the serialized length of your data, and
• you want to write to a strict bytestring (one big contiguous block of memory)

This last point notably may limit usage for serializing large data, depending on memory limitations. In most cases, we’ll use more memory than a buffering library such as mason.

So… why?

Well, bytezap will be slightly faster where it’s applicable, and the implementation is extremely simple. It’s a fun niche to fill, and it’s convenient for my binrep library.

Struct handling

We define even simpler parser & serializer types which can only handle “C struct”-like types, where there is only one constructor and all fields have known length at compile time. The way these work, GHC is pretty much guaranteed to generate the fastest code possible. These are very experimental, but see binrep for example usage.

Non-features

Serialize to ByteString (pinned byte arrays) only

No ShortByteStrings, no writing directly to handles.

(One may support writing to ShortByteStrings (unpinned byte arrays) by doing a bunch of class indirection. But it’s a lot of extra work for a use case that I don’t see as very popular at all. Check the Git history for an early implementation.)

License

Provided under the MIT license. See LICENSE for license text.