README.md

metadata

license: llama2
language:
  - en
tags:
  - tokenizer

A copy of Meta's llama tokenizer, with three tokens added to mask PII:

• |||EMAIL_ADDRESS|||
• |||PHONE_NUMBER|||
• |||IP_ADDRESS|||