Local ES ↔ EN Speech Interpreter

A bidirectional Spanish–English speech interpreter for medical and legal settings that runs entirely on your own machine. No audio and no text ever leaves the computer, which is what makes it usable where patient or client confidentiality is not negotiable.

On top of the real-time path there is an agent layer that measures and improves the system's own output quality.

🇪🇸 Manual de uso completo en español: README.es.md

_{Live session in medico domain. Source audio is a publicly available medical interpreting practice video from Interpremed; each card shows the transcript and its translation, and the status bar reports the running counters.}

LIVE PATH (real-time, deliberately lean)
  audio → Silero VAD → faster-whisper → filters → local LLM → UI + JSONL log
   mic / system                  (anti-hallucination,      ▲
   (WASAPI loopback)              es/en restriction)  glossary + dialogue memory + case docs

AGENT LAYER (async / offline — never in the capture path)
  live:    self-repair by confidence · non-translation guard
  offline: glossary miner · fidelity judge · session orchestrator · session diagnostics

The core architectural decision

This is a cascade (ASR → LLM translation), not an end-to-end speech translation model such as SeamlessM4T. That is a deliberate trade-off, and it is the axis the whole design turns on.

An end-to-end model would cut latency, but it is closed to context injection. The entire value of this system is the opposite: forcing a mandatory glossary, a bilingual dialogue memory and retrieved passages from the case file into every single turn. Only an instructed LLM accepts that. Terminological control is the product; latency is a constraint to manage, so the pipeline is threaded to keep the capture loop free instead.

What it does

• Desktop app with continuous listening and near-live text (~1–2 s per turn).
• Captures from the microphone or from system audio (WASAPI loopback — a call, a video).
• Three registers with separate prompts: medical, legal, and conversational. The conversational one preserves slang, tone and profanity instead of formalizing it; the medical/legal prompt is a strict fidelity contract (no summarizing, no softening).
• Mandatory glossary per domain — the single biggest quality lever.
• Draft + revision: long utterances show up instantly as fragments, then get re-translated as one coherent whole once the turn closes.
• RAG over case documents (clinical history, case file) injected per turn.
• Practice mode for interpreters: records three versions of each utterance (source / machine translation / your own interpretation) so they can be scored against each other.
• Agents: live self-repair, glossary miner, LLM-as-judge for fidelity, session summary with domain-drift detection, and a post-session diagnostic tool.

Engineering highlights

The interesting part of this project is not the feature list — it is that the design decisions came out of measuring real sessions, and that measuring contradicted my intuitions more than once. Everything below is a real finding from the session logs.

The VAD was never closing turns. Symptoms pointed at the prompt, so I measured the log instead: 66 % of the UI cards were born from a blind length cut, not a real pause (475/723 turns). With a permissive threshold on noisy audio, Silero stayed stuck on "speech present" and a 20-second guillotine was doing all the cutting — mid-sentence. Fixing the threshold and the silence window brought length-cuts down to 15–17 %, making the guillotine a safety net again rather than the actual mechanism.

The anti-hallucination filter was silently deleting real speech, and twice as often in Spanish. It was all-or-nothing per fragment, so one trailing artifact could drop a whole 20-second utterance, and 9 of its 15 marker phrases were Spanish subtitle boilerplate. It now filters per Whisper segment, is asymmetric by length (long fragments are kept and flagged rather than dropped), and — most importantly — logs what it discards. Before that, the loss was invisible and therefore unmeasurable. The governing rule became show and flag rather than lose in silence.

2.3 % of the "translations" were not translations at all. Measured across 4,185 logged turns: the model would answer the speaker instead of translating (staying in the source language and inventing content), drift into Chinese mid-sentence ("I need some help" → "Necesito一些帮助" — the working LLM is a Chinese model), or pad the output with text nobody said. Nothing detected this, because the existing checks only looked at glossary terms and ASR confidence. There is now a guard that detects all three symptoms and re-translates.

And the dialogue memory turned out to be the cause, not just the victim. Retrying with the conversation memory in the prompt fails 2 out of 3 times; retrying without it succeeds 3 out of 3. The model imitates the bad translation that memory hands it as "already interpreted context" — which is why these failures appear in clusters (24 in a row in one session) rather than isolated. So the repair path drops the memory, and it also rewrites the poisoned memory entry so the failure stops propagating forward.

Tripling the glossary made the output worse. The glossary is described throughout this project as the main quality lever, so 72 mined terms were reviewed and applied — and the very next real session showed quality dropping. A forced glossary does not disambiguate senses: discharge ↔ alta is hospital discharge, so on "have you noticed any discharge?" the repair pass rewrote a correct translation ("secreción") into a wrong one ("alta"). Two more causes compounded it: two Spanish entries mapped to the same English word, guaranteeing a permanent "missing term", and the check compared exact substrings, so a plural counted as missing. The fix was to prune the ambiguous and merely-frequent terms, match tolerantly, and stop rewriting on single-word terms — flag them instead, since that is where polysemy lives. Re-running the same session went from 7 glossary rewrites to 0. Frequency turned out to be the wrong criterion for admitting a term; being unambiguous is the right one.

When the retry fails again, the card is flagged, not sealed as fixed. Showing a non-translation with a "corrected" badge is worse than not repairing it, because the interpreter would trust it.

Design principles

• The live path stays lean. Every expensive piece of intelligence runs asynchronously (worker threads) or offline (agents). Nothing blocking is ever added to audio capture — if the capture loop stalls, the device buffer overflows and speech is lost for good.
• Config-driven. Model, domain, language pair, VAD, RAG and every threshold live in config.yaml. No parameter is hardcoded.
• New capabilities ship disabled, so adding one cannot change existing behaviour.
• Human in the loop for anything structural: the agent proposes, the human approves. The glossary miner is the clearest case — of 117 mined candidates, several were confidently wrong (hierro → stool, where hierro means iron), because the miner had learned from turns the ASR had already transcribed badly. All of them came out plausible-looking.
• Fidelity over fluency. The medical/legal prompt forbids summarizing, softening or paraphrasing, and that contract is never relaxed to make output read better.

Tests

Deterministic regression scripts with fake VAD and fake ASR — no GPU, no LLM, no real audio:

python tests\test_captura.py        # 66 checks: ASR filters, cutting, grouping, guard, agents
python tests\test_gui_tarjetas.py   # 18 checks: card accumulation and consolidation rendering

They are not a formal suite and they deliberately do not pin values that are meant to be tuned. VAD thresholds cannot be validated by any test — only by a real session, which is what src/agents/session_check.py exists for: it contrasts each signal of a finished session against its threshold and says what to adjust.

Known limits

Stated plainly, because a portfolio piece that hides its limits is not informative.

• It does not know who is speaking. Diarization was evaluated and deliberately dropped: a heavy dependency plus per-fragment latency, for a benefit that only applies with 3+ voices in one stream. The consequence is an accepted ceiling — cross-speaker references ("what he said") stay unresolved, and a silence-based cut can merge two speakers into one fragment. No prompt and no bigger model fixes this.
• Simultaneous overlapping voices degrade transcription. The system degrades gracefully rather than freezing, but the text is imprecise.
• The LLM can still fail to translate. The guard catches and retries, but this is a model limitation, not a pipeline bug.
• Quality depends heavily on source audio; conferencing apps compress and gate the signal.

Requirements

• Python 3.10+ (tested on 3.14).
• NVIDIA GPU recommended. On Blackwell (RTX 50xx) install torch cu128.
• Ollama running:

  ollama pull qwen2.5:14b-instruct    # translator LLM
  ollama pull bge-m3                  # embeddings (only if you enable RAG)
  

• Windows for system-audio capture (WASAPI loopback via PyAudioWPatch). Microphone capture works on any platform through sounddevice.

Setup

python -m venv .venv
.venv\Scripts\activate               # Linux/macOS: source .venv/bin/activate
pip install -r requirements.txt
.venv\Scripts\pythonw.exe gui.py     # desktop app
python main.py                       # push-to-talk CLI

First run downloads the Whisper (large-v3) and Silero VAD weights.

Offline agents:

python -m src.agents.glossary_miner          # propose glossary entries for review
python -m src.agents.judge --limit 25        # score fidelity -> eval_report.md
python -m src.agents.session_orchestrator    # session summary + domain drift
python -m src.agents.session_check --last 30 # diagnose the session you just ran

A note on privacy

Session transcripts are excluded from this repository by design (.gitignore): they contain real speech, and in this domain that means clinical or legal content. The example case document under docs/ is fictional. If you run this, your own transcript.jsonl stays on your machine — which is the point of the whole project.

Disclaimer

In real medical or legal settings, translation fidelity carries liability. This tool assists; it does not replace a certified interpreter or professional in high-stakes scenarios.

License

MIT — see LICENSE.