tokenize

Simple tokenizer for English text

https://github.com/haskell/tokenize

LTS Haskell 23.1:0.3.0.1
Stackage Nightly 2024-12-21:0.3.0.1
Latest on Hackage:0.3.0.1

See all snapshots tokenize appears in

BSD-3-Clause licensed by Grzegorz Chrupała
Maintained by Andreas Abel
This version can be pinned in stack with:tokenize-0.3.0.1@sha256:146423dea3f30146cdce0e4f386854568fc11ea2f8c284745544148926dc87f2,1495

Module documentation for 0.3.0.1

Depends on 3 packages(full list with versions):
base, split, text

Simple tokenizer for English text.

Changes

0.3.0.1 – 2024-08-04

  • Update benchmark to criterion >= 1.
  • Tested with GHC 7.10 - 9.10 (Cabal) and 8.2 - 9.8 (Stack).