Nemo Curator

GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.

Originally published by orchestra. Licensed under MIT. View original: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/05-data-processing/nemo-curator

Anthropic and OpenAI are registered trademarks of their respective owners. skill.ski is not affiliated with or endorsed by Anthropic or OpenAI.

Add the free skill.ski MCP server to your .mcp.json or equivalent config file:

{
  "mcpServers": {
    "skill-ski-free": {
      "url": "https://mcp.skill.ski/free",
      "type": "http"
    }
  }
}

Once connected, this Oski is available as a callable tool at your runtime. No paywall. No sign-in required for free-tier Oskis.