Tokenizers

tokenizers Module

Module providing tokenizers for various languages.

yalign.tokenizers.get_tokenizer(language)

Get a tokenizer for a two character language code.