hinbox

hinbox is a flexible, domain-configurable entity extraction system designed for historians and researchers. It processes historical documents, academic papers, news articles, and book chapters to extract structured information about people, organizations, locations, and events. Originally designed for Guantánamo Bay media coverage analysis, it now supports any historical or research domain through a simple configuration system.

🎯 Key Features

Research-Focused: Designed for historians, academics, and researchers
Flexible Sources: Process historical documents, academic papers, news articles, book chapters
Domain-Agnostic: Configure for any historical period, region, or research topic
Multiple AI Models: Support for cloud models (Gemini, OpenAI, Anthropic, OpenRouter) and local models (Ollama) via direct SDK integration
Entity Extraction: Automatically extract people, organizations, locations, and events
Smart Deduplication: RapidFuzz lexical blocking + embedding similarity with per-entity-type thresholds
Merge Dispute Agent: Second-stage LLM arbitration for ambiguous gray-band entity matches
5-Layer Canonical Name Selection: Deterministic scoring picks the best display name across aliases, penalizing acronyms and generic phrases
Profile Versioning: Track how entity profiles evolve as new sources are processed
Profile Grounding: Citation-backed claim verification checks that profile text is supported by source articles
Extraction Quality Controls: Deterministic QC with automatic retry when severe issues (zero entities, high drop rates) are detected
Extraction Caching: Persistent sidecar cache avoids redundant LLM calls when re-processing unchanged articles
Parallel Pipeline: Concurrent article extraction with LLM rate limiting and batched embedding computation
Modular Engine: src/engine coordinates article processing, extraction, merging, and profile versioning so new domains can reuse the same pipeline
Privacy Mode: --local flag enforces local-only embeddings and disables all LLM telemetry callbacks
Web Interface: FastHTML-based "Archival Elegance" UI with confidence badges, alias display, tag pills, and profile version navigation
Easy Setup: Simple configuration files, no Python coding required

📸 Screenshots

Real-time processing logs showing entity extraction progress with detailed status updates

Main dashboard displaying research domains and extracted entity statistics

Organizations listing with search and filtering capabilities for research analysis

Detailed organization profile showing extracted information, sources, and version history

🚀 Quick Start

1. List Available Domains

just domains

2. Create a New Research Domain

just init palestine_food_history

3. Configure Your Research Domain

Edit the generated files in configs/palestine_food_history/:

config.yaml - Research domain settings and data paths
prompts/*.md - Extraction instructions tailored to your sources
categories/*.yaml - Entity type definitions relevant to your research

4. Process Your Sources

just process-domain palestine_food_history --limit 5

5. Explore Results

just frontend

📦 Installation

Prerequisites

Python 3.12+
uv (for dependency management)
Optional: Ollama (for local model support)
just (task runner)

Setup

Clone the repository:

git clone https://github.com/strickvl/hinbox.git
cd hinbox

Install dependencies:

# Cloud embeddings only (works on all platforms including Intel Mac):
uv sync

# With local embeddings (requires PyTorch — Linux/Apple Silicon/Windows):
uv sync --extra local-embeddings

Set up environment variables:

# Create a .env file (auto-loaded by just):
# The default cloud model is openai/gpt-6-luna:
echo 'OPENAI_API_KEY=your-openai-api-key' > .env
# Optional for local processing:
echo 'OLLAMA_API_URL=http://localhost:11434/v1' >> .env

Set up local model (optional, requires Ollama):

# Pull the default local model (Qwen 2.5 32B, ~23GB download):
ollama pull qwen2.5:32b-instruct-q5_K_M

# Set a realistic context window (Ollama defaults are modest;
# Qwen 2.5 supports up to 131K tokens but more context = more VRAM).
# Add to your shell profile or systemd unit for the Ollama server:
export OLLAMA_CONTEXT_LENGTH=32768

You can override the default models without editing code by setting environment variables in your .env file:

# Override the local model (default: ollama/qwen2.5:32b-instruct-q5_K_M):
echo 'HINBOX_OLLAMA_MODEL=ollama/gemma3:27b' >> .env

# Override the cloud model (default: openai/gpt-6-luna, needs OPENAI_API_KEY).
# anthropic/... needs ANTHROPIC_API_KEY and `uv sync --extra anthropic`;
# gemini/... needs GEMINI_API_KEY
echo 'HINBOX_CLOUD_MODEL=anthropic/claude-haiku-5-5' >> .env

# Give one task its own model; the rest keep the default above. Tasks:
# RELEVANCE, EXTRACTION, MATCH, DISPUTE, PROFILE. The prefix decides where
# it runs (ollama/ = local). A --local run refuses cloud task models.
echo 'HINBOX_MATCH_MODEL=ollama/qwen3.6:35b' >> .env
echo 'HINBOX_PROFILE_MODEL=openai/gpt-6-luna' >> .env

# Ollama models get reasoning_effort="none" (thinking off) by default;
# set this to "" to keep a model's own default, or e.g. "high":
echo 'HINBOX_OLLAMA_REASONING_EFFORT=' >> .env
# With thinking on, Ollama calls get at least this many output tokens
# (default 8192) so the model can finish thinking and still answer:
echo 'HINBOX_OLLAMA_THINKING_MAX_TOKENS=8192' >> .env
# Ollama silently cuts the start off a prompt longer than the model's
# context (num_ctx), so Hinbox refuses such prompts. It reads num_ctx from
# the Ollama server; set this only if the server doesn't report it:
echo 'HINBOX_OLLAMA_NUM_CTX=32768' >> .env

Verify installation:
```
just domains
```

📚 Research Domain Examples

History of Food in Palestine

just init palestine_food_history
# Edit configs/palestine_food_history/ to focus on:
# - People: farmers, traders, cookbook authors, anthropologists
# - Organizations: agricultural cooperatives, food companies, research institutions
# - Events: harvests, famines, recipe documentation, cultural exchanges
# - Locations: villages, markets, agricultural regions, refugee camps

Soviet-Afghan War (1980s)

just init afghanistan_1980s
# Configure for:
# - People: military leaders, diplomats, journalists, mujahideen commanders
# - Organizations: military units, intelligence agencies, NGOs, tribal groups
# - Events: battles, negotiations, refugee movements, arms shipments
# - Locations: provinces, military bases, refugee camps, border crossings

Medieval Trade Networks

just init medieval_trade
# Set up for:
# - People: merchants, rulers, scholars, travelers
# - Organizations: trading companies, guilds, monasteries, courts
# - Events: trade agreements, diplomatic missions, market fairs
# - Locations: trading posts, cities, trade routes, ports

🛠 Advanced Usage

Processing Historical Sources

# Process with different options
just process --domain afghanistan_1980s --limit 20 --verbose
just process-domain palestine_food_history --limit 10 --relevance-check

# Use local models (requires Ollama) - useful for sensitive historical research
just process --domain medieval_trade --local

# Force reprocessing when you update your configuration
just process --domain afghanistan_1980s --force-reprocess

Web Interface

just frontend

Explore extracted entities at http://localhost:5001

Prompts per model

A prompt that suits one model can hurt another: the profile-edit prompt tuned for Qwen3.6 lifted the local models but made GPT-6 Luna skip real new facts. So every prompt file can have variants named <prompt>.<variant>.md next to the default <prompt>.md. For each prompt, the pipeline uses the first file that exists:

a family listed under prompt_families in the domain's config.yaml whose name parts appear in the model name (qwen: [qwen] makes ollama/qwen3.6:35b use relevance.qwen.md);
<prompt>.local.md for ollama/ models, <prompt>.cloud.md for the rest;
<prompt>.md.

This covers relevance, the four extraction prompts and the profile prompts; the merge check builds its prompt in code. Add a variant only when just eval show --grid shows it beating the default for that model by 3 or more cases. For example, over the 245 profile-update cases (train, dev and test together), profile_edit.cloud.md passes 225 on Luna and 215 on Haiku, against 213 and 191 for the plain profile_edit.md: the plain prompt turns passing mentions into 3 to 5 sentences.

Evaluating Models

The scoreboard grades any model on your hand-labelled answer keys, so you can compare cloud, frontier and local models (or later, optimized prompts) on equal terms.

# Score a model on the relevance labels or the merge labels
just eval relevance --model openai/gpt-6-luna
just eval merge --model ollama/qwen3.6:35b-a3b --split test

# Show every run so far, or the latest score for each model and prompt file
just eval show
just eval show --grid

# Score a prompt variant the pipeline wouldn't pick for this model
# (relevance.qwen.md, profile_edit.local.md, ...; 'default' = the plain file)
just eval profile-update --model openai/gpt-6-luna --split test --prompt-variant local

# Pick hard look-alike entity pairs to label (writes merge_candidates_v2.jsonl)
just eval-mine

# Grow long profiles by replaying well-covered entities' articles in date order
just eval-replay --list                          # article counts only, no model calls
just eval-replay --model openai/gpt-6-luna --max-articles 40

# Turn those snapshots into profile-update test cases
just eval-cases --model openai/gpt-6-luna

# Run the pipeline's profile update on those cases and score it
just eval profile-update --model openai/gpt-6-luna --split dev
just eval profile-update --model openai/gpt-6-luna --split dev --approach rewrite

Gold files live in {output_dir}/eval/. Each run writes its per-item predictions to eval/runs/ and adds one row to eval/scoreboard.jsonl. Items are split 60/20/20 into train/dev/test by a stable hash, so anything tuned on train is never graded on test. Results report balanced accuracy next to the "always give the common answer" baseline, because the label sets are lopsided.

just eval-replay prepares data for a profile-update eval. The entity database has no profile built from more than three articles, so it takes eighteen entities the corpus covers heavily (Nashiri, Judge Pohl, Camp Justice, the USS Cole bombing, ...), creates a profile from the first article mostly about each one, then runs the pipeline's own update_profile on each later article. It saves the profile after every step to eval/profile_replay/<model>/<entity>/step_NNN.json, and resumes where it stopped.

just eval-cases turns those snapshots into test cases, each one an existing profile plus an article. There are three kinds: an article that only mentions the entity in passing (a good update keeps everything and changes little), the real next article (a good update adds the new facts with its citation and keeps the rest), and a copy of the next article with one year changed (a good update keeps both years, each with its own source). Cases are split into train/dev/test by entity (6/6/6 for the eighteen replay entities, with every entity type in each split), so a tuned prompt never sees a test entity. Disputed cases come from a word-overlap heuristic and are marked needs_review until a person confirms the two sentences make the same claim.

just eval profile-update runs the pipeline's update_profile on each case and scores the result without a model. A case fails if the profile shrinks by more than 20%, if fewer than 90% of its cited sentences survive (reworded sentences still count as kept), or if it cites an id that is neither an old source nor the new article. Each kind also has its own check: new-fact cases must cite the article, passing-mention cases may add at most two sentences, and disputed cases must keep the old year with its old source and state the new year with the new article. The scoreboard also shows how much old text was kept word for word, because a good update shouldn't rewrite text it had no reason to touch. In the full replay, 40 single updates cut a profile by more than a fifth (one went from 31k characters to 6k), and every one passed the pipeline's own QC.

The pipeline's update_profile uses an edit-list updater (--approach rewrite scores the old whole-profile rewrite instead). Rather than writing the whole profile again, the model returns a short list of edits: add these sentences after this existing sentence (or under this heading, or at the end), or replace this one sentence with a sharper version. Code applies the edits to the old text, so a sentence the model never mentions stays exactly as it was, and a replaced sentence keeps its citations. The prompt (profile_edit.cloud.md for cloud models, profile_edit.local.md for local ones) also says to keep both versions, each with its source, when the article contradicts the profile, and to add little or nothing from an article that only mentions the entity in passing. On the dev and test cases it passes 0.89 on Luna against 0.29 for the rewrite, in about 2s per update instead of 21s. Set HINBOX_PROFILE_UPDATE_APPROACH=rewrite to run the pipeline with the old rewrite.

DSPy experiments live in experiments/dspy_opt/, in their own environment, so DSPy never enters Hinbox's main dependencies, and LiteLLM is never installed. just dspy-merge --model <model> --optimizer mipro optimises the merge check for one model and puts the result on the same scoreboard. just dspy-profile --model <model> --reflection-model openai/gpt-6-luna runs GEPA on the edit-list profile updater.

Data Management

# Check processing status
just check

# Reset processing status
just reset

# View available domains
just domains

📂 Project Structure

configs/
├── guantanamo/        # Example domain shipped with the project
├── template/          # Starter files copied by `just init`
└── README.md          # Domain configuration walkthrough

src/
├── process_and_extract.py  # CLI entry point — parallel producer/consumer pipeline
├── engine/
│   ├── article_processor.py   # Relevance → extraction → QC retry orchestration
│   ├── extractors.py          # Unified cloud/local entity extraction
│   ├── mergers.py             # Lexical blocking → embedding similarity → match check → dispute
│   ├── merge_candidates.py    # Lexical blocking + embedding similarity search
│   ├── merge_evidence.py      # Deterministic evidence text + batch embedding
│   ├── match_checker.py       # LLM-based match verification
│   ├── merge_dispute_agent.py # Second-stage arbitration for gray-band matches
│   ├── profiles.py            # VersionedProfile history management
│   └── relevance.py           # Domain-specific relevance filtering
├── frontend/               # FastHTML "Archival Elegance" UI
│   ├── routes/             # Modular route handlers (home, people, orgs, locations, events)
│   ├── components.py       # Reusable UI building blocks (badges, version selectors, tags)
│   ├── data_access.py      # Centralised Parquet data loading
│   └── static/styles.css   # CSS variables, fonts, layout
├── utils/
│   ├── embeddings/         # EmbeddingManager, cloud/local providers, similarity helpers
│   ├── cache_utils.py      # Thread-safe LRU cache and stable hashing
│   ├── extraction_cache.py # Persistent sidecar cache for extraction results
│   ├── name_variants.py    # Deterministic name normalisation, acronym detection, canonical scoring
│   ├── processing_status.py # Sidecar JSON tracker (replaces in-Parquet status)
│   ├── outcomes.py         # PhaseOutcome structured result objects
│   └── quality_controls.py # Extraction QC, profile QC, and profile grounding verification
├── config_loader.py        # Domain config loader (incl. per-type thresholds, lexical blocking)
├── dynamic_models.py       # Domain-driven Pydantic model factories
├── constants.py            # Model defaults, embedding settings, thresholds, privacy controls
├── logging_config.py       # Rich-based structured logging with colour-coded decision lines
└── exceptions.py           # Custom exception types used across the pipeline

tests/
├── embeddings/                         # Embedding manager, cloud provider, config integration
├── test_canonical_name.py              # 5-layer canonical name scoring and rekey-on-merge
├── test_cli_privacy_mode.py            # --local flag enforces local embeddings
├── test_domain_paths.py                # Domain-specific path resolution and batch writes
├── test_entity_merger_merge_smoke.py   # Embedding-based merge smoke tests
├── test_entity_merger_similarity.py    # Similarity scoring, lexical blocking, fingerprints
├── test_extraction_cache.py            # Sidecar cache key determinism and roundtrip
├── test_extraction_retry.py            # QC-triggered retry logic and repair hints
├── test_llm_multiple_tool_calls.py     # Instructor multi-tool-call recovery
├── test_merge_dispute_agent_routing.py # Gray-band routing and dispute decisions
├── test_name_variants.py               # Name normalisation, acronyms, equivalence groups
├── test_profile_grounding.py           # Citation extraction and grounding verification
├── test_profile_versioning.py          # Versioned profile regression tests
└── test_frontend_versioning.py         # UI behaviour for profile history

data/
├── guantanamo/        # Default domain data directory (created locally)
└── {domain}/          # Additional domains maintain their own raw/entity data

🔧 Configuration

Domain Configuration

Each domain has its own configs/{domain}/ directory with:

config.yaml - Main settings:

domain: "palestine_food_history"
description: "Historical analysis of Palestinian food culture and agriculture"
data_sources:
  default_path: "data/palestine_food_history/raw_sources/historical_sources.parquet"
output:
  directory: "data/palestine_food_history/entities"

categories/*.yaml - Entity type definitions:

person_types:
  player:
    description: "Professional football players"
    examples: ["Lionel Messi", "Cristiano Ronaldo"]

prompts/*.md - Extraction instructions (plain English!):

You are an expert at extracting people from historical documents about Palestinian food culture.
Focus on farmers, traders, cookbook authors, researchers, and community leaders...

Data Format

Historical sources should be in Parquet format with columns:

title: Document/article title
content: Full text content
url: Source URL (if applicable)
published_date: Publication/creation date
source_type: "book_chapter", "journal_article", "news_article", "archival_document", etc.

🏗 Architecture

Processing Pipeline

Configuration Loading: Read domain-specific settings
Source Loading: Process historical documents in Parquet format
Relevance Filtering: Domain-specific content filtering for research focus
Parallel Extraction: Concurrent article + entity-type extraction with LLM rate limiting (ThreadPoolExecutor workers, bounded semaphore for API calls)
Extraction Caching: Persistent sidecar cache keyed on content hash, model, prompt, and schema — skips redundant LLM calls on re-runs
Quality Controls: Deterministic QC validates extraction output (required fields, name normalisation, within-article dedup) with automatic retry on severe flags
Smart Deduplication: Lexical blocking pre-filter → batched embedding similarity → evidence-first merge cost structure (cheap checks before expensive LLM calls)
Merge Dispute Resolution: Gray-band matches (similarity near threshold with low confidence) get a second-stage LLM arbitration via MergeDisputeAgent
Canonical Name Selection: 5-layer scoring picks the best display name when entities merge, penalizing acronyms and generic phrases
Profile Generation: Create comprehensive entity profiles with citation-backed claims and automatic versioning
Profile Grounding: Post-processing verification that profile claims are supported by cited source articles
Persistence: Batched Parquet writes per entity type (avoiding write amplification), sidecar JSON for processing status

Engine Modules

ArticleProcessor orchestrates relevance checks, extraction dispatch (with QC retry), and per-article metadata aggregation (src/engine/article_processor.py)
EntityExtractor unifies cloud and local model calls using domain-specific Pydantic schemas (src/engine/extractors.py)
EntityMerger pre-filters with RapidFuzz lexical blocking, compares batched embeddings, calls match-checkers, routes gray-band cases to the dispute agent, and updates persisted Parquet rows (src/engine/mergers.py)
MergeDisputeAgent provides second-stage structured LLM analysis for ambiguous merge/skip decisions (src/engine/merge_dispute_agent.py)
VersionedProfile and helper functions maintain profile history for each entity (src/engine/profiles.py)

Key Features

Domain-Agnostic: Easy to configure for any topic
Multiple AI Models: Cloud (Gemini) and local (Ollama) support
Privacy Mode: --local flag forces local embeddings and disables all LLM telemetry
Smart Processing: Automatic relevance filtering, caching, and multi-layer deduplication
Profile Versioning: Track entity profile changes over time with full version history
Profile Grounding: Citation-backed claim verification for generated profiles
Modern Interface: FastHTML "Archival Elegance" theme with confidence badges, aliases, tag pills, and version navigation
Robust Pipeline: Structured PhaseOutcome error handling, quality controls, extraction retry, and progress tracking
Structured Logging: Colour-coded decision lines (NEW, MERGE, SKIP, DISPUTE, DEFER) for pipeline transparency

Development

Testing

# Run all tests
just test

# Run specific test files
just test -k test_profile_versioning
just test tests/test_entity_merger_similarity.py

CI runs lint and tests automatically on every PR (.github/workflows/test.yml). The test suite covers embedding similarity, lexical blocking, per-type threshold resolution, entity merger behavior, merge dispute agent routing, extraction caching, extraction retry logic, canonical name selection, name variant detection, profile grounding, profile versioning, privacy mode enforcement, and frontend components — all without requiring API keys or GPU.

Code Quality

# Format code
just format

# Lint code
just lint

# Both together
just check-code

# Fail if any tracked .py file is over 1000 lines
just check-file-length

# Run exactly what CI runs (recommended before pushing)
just ci

🤝 Contributing

Contributions welcome! Areas of interest:

New domain templates and examples
Additional language model integrations
Enhanced web interface features
Performance optimizations

📄 License

MIT License - see LICENSE file for details.

🙋 Support

For questions about:

Configuration: See configs/README.md
Setup: Check installation steps above
Usage: Try just or just --list
Issues: Open a GitHub issue

Built for: Historians, researchers, and academics working with large document collections

Built with: Python, Pydantic, FastHTML, Instructor, OpenAI SDK, RapidFuzz, Jina Embeddings, Rich

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
assets		assets
configs		configs
experiments/dspy_opt		experiments/dspy_opt
scripts		scripts
src		src
tests		tests
.gitignore		.gitignore
.python-version		.python-version
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
README.md		README.md
justfile		justfile
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation