Skip to content

Latest commit

 

History

History
100 lines (79 loc) · 6.31 KB

ARCHITECTURE.md

File metadata and controls

100 lines (79 loc) · 6.31 KB

TokenMaxxing Architecture & Implementation Roadmap

1. Project Vision

TokenMaxxing is a production-grade open-source platform designed to maximize Claude Code efficiency by minimizing token consumption while preserving or enhancing output quality. It moves beyond naive token counting into intelligent context compression, deduplication, and semantic pruning.

2. Research & Architecture Analysis

Before designing TokenMaxxing, we analyzed several existing tokenizers:

  • Jellyfishboy/claude-tokenizer:
    • Architecture: Rust-based offline token estimation mapping Anthropic tokenization schemes.
    • Strengths: High performance, no API overhead, offline capability.
    • Weaknesses: Static mapping; requires updates if Anthropic changes algorithms.
  • javirandor/anthropic-tokenizer:
    • Architecture: Experimental API-probing and approximation.
    • Strengths: Empirically accurate for newer, closed models (like Claude 3).
    • Weaknesses: Network overhead, high latency, brittle to API changes.
  • mideind/Tokenizer:
    • Architecture: Rule-based linguistic tokenizer (primarily for Icelandic text normalization).
    • Strengths: High linguistic precision, structured token classification.
    • Weaknesses: Not suited for sub-word LLM tokenization or general-purpose English LLM prompts.
  • tiktoken / SentencePiece / Modern BPE:
    • Architecture: Byte-Pair Encoding (BPE) and Unigram models. Tiktoken is highly optimized in Rust for OpenAI models.
    • Strengths: Extremely fast, industry standard for BPE.

TokenMaxxing Innovation: TokenMaxxing will use a high-performance Rust core (similar in spirit to tiktoken and claude-tokenizer) for offline token counting, combined with semantic context compression, deduplication heuristics, and AST-aware repository analysis.

3. High-Level Architecture

TokenMaxxing is built on a modular architecture:

  1. Core (Rust): High-performance engine for token estimation, compression algorithms, and repository analysis.
  2. CLI (Rust): The tokenmaxxing binary for executing optimizations, stats, and reports directly from the terminal.
  3. Bindings: Python (via PyO3) and TypeScript (via N-API / WebAssembly) wrappers to allow seamless integration into varied developer environments and agent workflows.

Modules Breakdown

3.1 Token Engine

The engine keeps the different levels of certainty apart. Every TokenCount carries a CountKind that states how it was produced.

  • Fast Mode (CountKind::Estimated): Character ratio heuristic of roughly 3.5 characters per token. Used for massive repositories.
  • Accurate Mode (CountKind::Approximate): Structural emulation of byte-pair encoding. Whitespace is attached to the following word, punctuation is split per character, and long words are split into sub-word pieces. Without a vocabulary this stays an approximation.
  • Exact Mode (CountKind::ExactLocal): Backed by a LocalTokenizer that ships a real vocabulary. Enabled through TokenEngine::with_tokenizer.
  • Provider Mode (CountKind::ProviderApi): Backed by a TokenCountProvider such as a future Anthropic or OpenAI adapter. The core crate stays free of vendor SDKs and HTTP clients.
  • Streaming Tokenization: Decodes UTF-8 incrementally, carries incomplete multi-byte sequences over read boundaries and keeps pieces aligned across chunk boundaries, so streaming returns the same count as counting the whole text at once.
  • Pricing: Cost estimation takes the rates from the caller, no model prices are hardcoded.

3.2 Prompt Optimizer

  • Deduplication Engine: Identifies duplicate instructions across concatenated prompt files.
  • Minifiers: Specialized parsers to minify JSON, XML, Markdown, and YAML without breaking syntax.

3.3 Context Compressor

  • Semantic Pruning: Uses lightweight embeddings or TF-IDF heuristics to rank and remove "dead" context (e.g., boilerplate code irrelevant to the prompt).
  • Summarization: Progressively collapses older or less relevant instruction chunks.

3.4 Repository Intelligence

  • AST Analysis: Uses tree-sitter to parse code structure and selectively prune non-essential blocks (like stripping function bodies from unmodified files when only signatures are needed).
  • Bloat Detection: Flags repetitive templates and bloated docs.

3.5 Agent Optimization Layer

  • Workflow Analyzer: Monitors repetitive tool calls (e.g., repeated cat or grep yielding same outputs) and caches/summarizes them.

4. Implementation Roadmap

Phase 1: Foundation (v0.1.0)

  • Research & Architecture Design (Complete)
  • Repository Setup: Cargo workspace, module skeletons.
  • Token Engine: Implement Fast and Accurate estimation modes in Rust.
  • CLI Base: Basic CLI structure (count, stats commands).
  • Unit Tests and CI/CD pipelines setup.

Phase 2: Optimization Engines (v0.2.0)

  • Prompt Optimizer: Implement Markdown, JSON, XML minification.
  • Prompt Optimizer: Instruction deduplication logic.
  • CLI Integration: Implement optimize and benchmark commands.
  • Python and TypeScript bindings for the Token Engine and Prompt Optimizer.

Phase 3: Context & Repository Intelligence (v0.3.0)

  • Context Compressor: Semantic chunking and ranking algorithms.
  • Repository Intelligence: tree-sitter integration for code structure analysis and bloated file detection.
  • CLI Integration: Implement compress and analyze commands.

Phase 4: Agent Workflow & Polish (v1.0.0)

  • Agent Optimization Layer: Hooks and analyzers for agent workflows.
  • Prompt Maxxing Mode: Experimental intent-preserving prompt rewriting.
  • Finalization of all docs, benchmarks (against tiktoken & claude-tokenizer), and release notes.

5. Technology Stack Details

  • Language: Rust (Edition 2021)
  • CLI Framework: clap
  • Serialization: serde, serde_json
  • Parsing: tree-sitter
  • Concurrency: rayon for parallel repository scanning
  • Python Bindings: pyo3, maturin
  • TypeScript Bindings: napi-rs

6. Development Rules Enforced

  • No placeholder implementations.
  • Full test coverage.
  • Public APIs strictly documented.
  • Benchmarks run automatically on CI.