TokenMaxxing is a production-grade open-source platform designed to maximize Claude Code efficiency by minimizing token consumption while preserving or enhancing output quality. It moves beyond naive token counting into intelligent context compression, deduplication, and semantic pruning.
Before designing TokenMaxxing, we analyzed several existing tokenizers:
- Jellyfishboy/claude-tokenizer:
- Architecture: Rust-based offline token estimation mapping Anthropic tokenization schemes.
- Strengths: High performance, no API overhead, offline capability.
- Weaknesses: Static mapping; requires updates if Anthropic changes algorithms.
- javirandor/anthropic-tokenizer:
- Architecture: Experimental API-probing and approximation.
- Strengths: Empirically accurate for newer, closed models (like Claude 3).
- Weaknesses: Network overhead, high latency, brittle to API changes.
- mideind/Tokenizer:
- Architecture: Rule-based linguistic tokenizer (primarily for Icelandic text normalization).
- Strengths: High linguistic precision, structured token classification.
- Weaknesses: Not suited for sub-word LLM tokenization or general-purpose English LLM prompts.
- tiktoken / SentencePiece / Modern BPE:
- Architecture: Byte-Pair Encoding (BPE) and Unigram models. Tiktoken is highly optimized in Rust for OpenAI models.
- Strengths: Extremely fast, industry standard for BPE.
TokenMaxxing Innovation: TokenMaxxing will use a high-performance Rust core (similar in spirit to tiktoken and claude-tokenizer) for offline token counting, combined with semantic context compression, deduplication heuristics, and AST-aware repository analysis.
TokenMaxxing is built on a modular architecture:
- Core (Rust): High-performance engine for token estimation, compression algorithms, and repository analysis.
- CLI (Rust): The
tokenmaxxingbinary for executing optimizations, stats, and reports directly from the terminal. - Bindings: Python (via PyO3) and TypeScript (via N-API / WebAssembly) wrappers to allow seamless integration into varied developer environments and agent workflows.
The engine keeps the different levels of certainty apart. Every TokenCount carries a CountKind that states how it was produced.
- Fast Mode (
CountKind::Estimated): Character ratio heuristic of roughly 3.5 characters per token. Used for massive repositories. - Accurate Mode (
CountKind::Approximate): Structural emulation of byte-pair encoding. Whitespace is attached to the following word, punctuation is split per character, and long words are split into sub-word pieces. Without a vocabulary this stays an approximation. - Exact Mode (
CountKind::ExactLocal): Backed by aLocalTokenizerthat ships a real vocabulary. Enabled throughTokenEngine::with_tokenizer. - Provider Mode (
CountKind::ProviderApi): Backed by aTokenCountProvidersuch as a future Anthropic or OpenAI adapter. The core crate stays free of vendor SDKs and HTTP clients. - Streaming Tokenization: Decodes UTF-8 incrementally, carries incomplete multi-byte sequences over read boundaries and keeps pieces aligned across chunk boundaries, so streaming returns the same count as counting the whole text at once.
- Pricing: Cost estimation takes the rates from the caller, no model prices are hardcoded.
- Deduplication Engine: Identifies duplicate instructions across concatenated prompt files.
- Minifiers: Specialized parsers to minify JSON, XML, Markdown, and YAML without breaking syntax.
- Semantic Pruning: Uses lightweight embeddings or TF-IDF heuristics to rank and remove "dead" context (e.g., boilerplate code irrelevant to the prompt).
- Summarization: Progressively collapses older or less relevant instruction chunks.
- AST Analysis: Uses
tree-sitterto parse code structure and selectively prune non-essential blocks (like stripping function bodies from unmodified files when only signatures are needed). - Bloat Detection: Flags repetitive templates and bloated docs.
- Workflow Analyzer: Monitors repetitive tool calls (e.g., repeated
catorgrepyielding same outputs) and caches/summarizes them.
- Research & Architecture Design (Complete)
- Repository Setup: Cargo workspace, module skeletons.
- Token Engine: Implement Fast and Accurate estimation modes in Rust.
- CLI Base: Basic CLI structure (
count,statscommands). - Unit Tests and CI/CD pipelines setup.
- Prompt Optimizer: Implement Markdown, JSON, XML minification.
- Prompt Optimizer: Instruction deduplication logic.
- CLI Integration: Implement
optimizeandbenchmarkcommands. - Python and TypeScript bindings for the Token Engine and Prompt Optimizer.
- Context Compressor: Semantic chunking and ranking algorithms.
- Repository Intelligence:
tree-sitterintegration for code structure analysis and bloated file detection. - CLI Integration: Implement
compressandanalyzecommands.
- Agent Optimization Layer: Hooks and analyzers for agent workflows.
- Prompt Maxxing Mode: Experimental intent-preserving prompt rewriting.
- Finalization of all docs, benchmarks (against
tiktoken&claude-tokenizer), and release notes.
- Language: Rust (Edition 2021)
- CLI Framework:
clap - Serialization:
serde,serde_json - Parsing:
tree-sitter - Concurrency:
rayonfor parallel repository scanning - Python Bindings:
pyo3,maturin - TypeScript Bindings:
napi-rs
- No placeholder implementations.
- Full test coverage.
- Public APIs strictly documented.
- Benchmarks run automatically on CI.