Sentencepiece

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

Originally published by orchestra. Licensed under MIT. View original: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/02-tokenization/sentencepiece

Anthropic and OpenAI are registered trademarks of their respective owners. skill.ski is not affiliated with or endorsed by Anthropic or OpenAI.

Add the free skill.ski MCP server to your .mcp.json or equivalent config file:

{
  "mcpServers": {
    "skill-ski-free": {
      "url": "https://mcp.skill.ski/free",
      "type": "http"
    }
  }
}

Once connected, this Oski is available as a callable tool at your runtime. No paywall. No sign-in required for free-tier Oskis.