gemma-curation-agents

Agentic helpers for Gemma curation. Each agent is a thin pipeline that reads external data (GEO, papers, Gemma's own annotation index), reasons with an LLM where useful, and writes a structured output a curator reviews. No agent writes to production Gemma directly — proposals land in a write-API queue (currently a mock; production endpoint TBD) and a human accepts / rejects / requests changes via the curation UI.

Layout

gemma_curation_agents/
  cli.py                       # `gca` entry point
  shared/                      # plumbing reused across agents
    llm.py                     # Anthropic client, structured output
    biolit.py                  # paper full-text via biolit + fetcher hooks
    gemma.py                   # gemma-mcp / gemmapy helpers
    keychain.py                # macOS keychain auto-loader
  agents/
    scrape_screen/             # GEO scrape screening (filters → project filter → LLM)
      filters.py               # 7 deterministic filters (RNA type, sample size, etc.)
      project_filter.py        # YAML-driven keyword filter per "project"
      screener.py              # LLM screener with structured output
      tf_lookup.py             # Lambert TF list + NCBI synonym aliases
      gemma_compare.py         # comparator vs existing Gemma curation
      ncbi_synonyms.py         # generic gene-symbol alias resolution
      build_tf_list.py         # builds tf_master.json (Lambert + DIOPT + synonyms)
    curation_proposer/         # proposes design + tags for loaded preboardings
      schemas.py               # shared Pydantic types
      tag_proposer.py
      design_proposer.py
      sample_assigner.py
      submitter.py             # POST proposal to the write-API
    audit/                     # audits already-curated experiments against guidelines
      pipeline.py              # orchestrates judges + silent comparison proposer
      schemas.py               # AuditFinding / AuditReport / disposition types
      snapshot.py              # extract audit targets from a Preboarding
      preboarding_context.py      # shared BM-characteristics + split-notice helpers
      submitter.py             # POST AuditReport to the write-API
      target_ids.py            # stable IDs for findings ↔ UI elements
      judges/
        forbidden_efc_judge.py     # deterministic forbidden-EFC table lookup
        fv_coverage_judge.py       # deterministic per-FV coverage check
        term_grounding_judge.py    # deterministic URI-presence check on tags
        factor_llm_judge.py        # LLM verdict per existing factor
        fv_llm_judge.py            # LLM verdict per existing FV
        tag_llm_judge.py           # LLM verdict per existing tag
      prompts/                 # one .md per LLM judge
  ontology/
    resolver.py                # protocol
    gemma_resolver.py          # gemmapy-backed
  local_api/     # FastAPI mock of the curation write-API
    server.py
    storage.py                 # SQLite
    design_schemas.py          # Pydantic for designs / tags / notes
    annotations.py             # /annotations/search catalog (loaded from Gemma java repo)
    categories.py              # /categories list (loaded from Gemma java repo)
    gemma_resources.py         # parsers for EFO.factor.categories.txt + valueStringToOntologyTermMappings.txt

data/
  tf_master.json               # built by `gca build-tf-list`
  epigenetic_exclusions.txt
scrape_out/
  REPORT_tf_pert_v2.md         # dispute analysis from the latest screen run
tests/unit/

Configuration & dependencies

All credentials resolve from the macOS Keychain at run start (never from .env files); a pre-set env var always wins, otherwise the Keychain entry of the same service name is used. Set an entry with security add-generic-password -s <NAME> -w <value>.

Software dependencies

dependency	required?	notes
Python 3.10+ venv (.venv)	required	pip install -e .
anthropic SDK	required (default LLM provider)	other providers planned; see the reuse plan
biolit (../biolit)	required for paper full text	PMC-first fetch chain + OA fetchers
FAISS + a BioLORD index	required for embedding resolver	build via the ontology-index step below
openai SDK	required for some resolver embeddings	needs OPENAI_API_KEY
Gemma REST + ontology host	required	preboarding + the ontology/annotation resolver
local_api (mock curation API)	for the write-back queue	./run_local.sh
zotero-mcp	OPTIONAL	only for reading human-attached paywalled PDFs (see below)

Credentials (Keychain service names)

service	required?	what for
ANTHROPIC_API_KEY (or anthropic)	required	the LLM
GEMMA_USERNAME, GEMMA_PASSWORD	required	Gemma is auth-gated; anon calls return empty
GEMMA_BASE_URL	required	Gemma REST target (the lab dev box)
OPENAI_API_KEY	required for resolver embeddings	embedding-based ontology grounding
NCBI_API_KEY	optional	higher NCBI/PubMed rate limits (pub_finder, biolit)
CORE_API_KEY	optional	the CORE open-access PDF fetcher
ZOTERO_USER_ID, ZOTERO_API_KEY	optional	only if using the Zotero PDF path

GEMMA_ONTOLOGY_URL / GEMMA_CURATION_URL must point the resolver at an annotation-index host (the runner refuses to launch otherwise; pass --ack-ontology-coverage to skip for paper-only passes).

Paper full text — and why Zotero is optional

The proposer reads the dataset's paper (methods carry strain / sex / age / stage detail). Resolution order is biolit first (PMC-first; reaches full text for ~83% of GSEs with no Zotero at all), then a paywalled fallback. The fallback is pluggable — you do not need Zotero:

• --no-paper — run preboarding-only (no paper context). Fully supported.
• BIOLIT_LOCAL_PDF_DIR (a.k.a. GEMMA_AGENTS_LOCAL_PDF_DIR) — point at a directory of PDFs you obtained any way you like; biolit's local_pdf fetcher uses them. This is the provider-neutral "provide the PDFs some other way" path.
• Zotero (optional) — reads human-attached PDFs back from a Zotero collection. It no-ops gracefully when zotero-mcp or ZOTERO_* creds are absent (falls through to biolit), so it's never required. The read is local-disk first (ZOTERO_STORAGE_DIR, default ~/Zotero/storage) so it needs no network; the Web API is only a fallback. Workflow: push paywalled papers to a collection with --pause-for-papers --papers-collection <KEY> for a human to fetch, then re-read on --resume. Controlled by --require-zotero (default on only when --paper is on — makes Zotero unavailability a loud stopper rather than a silent degrade); pass --no-require-zotero for a deliberately OA-only run, or --no-paper to skip papers entirely. (biolit has a separate internal Zotero fetcher gated by GEMMA_AGENTS_USE_ZOTERO.)

So a clean minimal setup is: Anthropic + Gemma + OpenAI creds, biolit, and either OA-only papers or a BIOLIT_LOCAL_PDF_DIR. Zotero is purely an optional convenience for the human-in-the-loop paywalled-PDF workflow.

Runtime / robustness knobs (env, all have safe defaults)

var	default	effect
GEMMA_STALL_TIMEOUT_S	900	stall watchdog: dump all thread stacks + abort if no progress this long (--stall-timeout). kill -USR1 <pid> dumps on demand
GEMMA_LLM_TIMEOUT_S / GEMMA_LLM_MAX_RETRIES	180 / 2	Anthropic client read/pool timeout + retries (prevents connection-pool deadlocks)
GEMMA_OPENAI_TIMEOUT_S	60	OpenAI embedding client timeout
GEMMA_PAPER_CACHE_DIR	~/Data/.../paper_context_cache	persistent per-GSE paper cache, reused across runs/arms; empty string disables
GEMMA_ZOTERO_READ_BUDGET_S / GEMMA_ZOTERO_READ_TRIES	45 / 3	wall-clock bound + retries on the Zotero read (a hang can't wedge the batch)
GEMMA_PUB_FINDER	1	pub_finder fallback when GEO links no paper (0 to disable)
GEMMA_REST_CACHE_DIR / GEMMA_REFRESH_CACHE	—	Gemma REST response cache

Quick start

Wrappers handle env loading from macOS keychain — preferred to avoid stale env-var traps:

# Mock curation API (terminal 1, leave running)
./run_local.sh
# auto-loads GEMMA_CURATION_API_KEY from keychain (or dev-token-123)
# auto-points GEMMA_RESOURCES_DIR at ~/Gemma if present
# pass --reload to enable uvicorn file-watcher when iterating on schemas
#   ./run_local.sh --reload

# Full TF-perturbation screen (terminal 2)
./run_screen.sh smoke --with-biolit         # 10-row sanity check
./run_screen.sh full  --with-biolit         # 584-row run, ~30 min

# Compare a screened TSV against existing Gemma curation
./run_compare.sh
./run_compare.sh path/to/screened.tsv tag

# Run a fresh proposal through the full pipeline
./run_propose.sh GSE277245

# Audit an already-curated experiment (or the hand-picked benchmark batch)
./run_audits.sh                                                  # batch: 10 experiments
.venv/bin/python -m scripts.reaudit_one GSE281069                # single

# Aggregate finalized-audit dispositions into a dismiss-rate report
.venv/bin/python -m scripts.eval_analysis.audit_dispositions

# Curation UI (separate repo)
cd ../gemma-ui && npm run dev

Direct CLI (no wrappers):

.venv/bin/python -m gemma_curation_agents.cli screen ...
.venv/bin/python -m gemma_curation_agents.cli propose-curation ...
.venv/bin/python -m gemma_curation_agents.cli mock-gemma serve --port 8080
.venv/bin/python -m gemma_curation_agents.cli build-tf-list

Ontology embedding indexes (optional)

Tier-1 of the resolver chain (HybridResolver with --embedding) runs dense retrieval over local FAISS HNSW indexes built from the working-set OBO files. Indexes are not shipped in the repo — each install builds them once, locally, into ~/Data/ontology_index/.

Most curators don't need this. Calibration-package review (setup.py + UI) requires zero embedding artefacts; the package carries all proposals pre-baked. Only install the indexes if you plan to run the agent locally with --embedding.

# Build all seven ontologies (UBERON, CL, MONDO, MP, HP, PATO, EFO).
# First run: ~15-30 min on a CPU laptop, ~1 GB disk.
.venv/bin/python scripts/build_ontology_index.py --all

# Or pick individual ontologies:
.venv/bin/python scripts/build_ontology_index.py UBERON CL

What happens on first run:

1. Downloads OBO files (~200 MB total) to ~/Data/ontology_owl/. Re-runs use the cache unless --force-download.
2. Downloads BioLORD-2023 encoder (~440 MB) to ~/.cache/huggingface/ on first embedding call.
3. Parses each OBO to ~/Data/ontology_index/<ONT>/terms.parquet.
4. Embeds every term + builds ~/Data/ontology_index/<ONT>/index.faiss. Single-threaded FAISS to avoid torch+faiss OpenMP contention on Intel macOS (see e432303 / docs/EMBEDDING_RESOLVER_SPEC.md).

Subsequent invocations cache-hit and finish in seconds.

Verify with the smoke probe:

.venv/bin/python scripts/smoke_embedding_resolver.py

Or the 30-case benchmark + scoring:

.venv/bin/python scripts/eval_embedding_resolver.py --resolver hybrid

Without the indexes, --embedding will print a warning and fall back silently to Tier-0 (CurationApiResolver + NcbiGeneResolver + xref-aware alignment). The agent stays functional — recall on the long-tail MBH-class case is slightly lower without Tier-1.

Pitfalls:

• ~/Data/ is the default; override with --owl-dir / --index-dir if you keep things elsewhere.
• HuggingFace Hub access is needed on first run for BioLORD — proxy / VPN issues are the common gotcha.
• Apple Silicon and Linux build cleanly; Intel macOS is fine too thanks to the single-thread pin but is slowest of the three.

Agents

scrape-screen

Three-stage screen of the listGEOData TSV:

1. Deterministic filters — RNA type, sample size, taxa, library strategy, blacklist, etc. Mirrors the legacy GEOscrape_post_processing buckets so curators can drop it in.
2. Project filter — YAML-driven keyword filter per topic ("tf_pert", "brain", etc.).
3. LLM screener — Anthropic Haiku reads the GEO record + paper full-text (when available via biolit) and emits a structured verdict: yes / no / borderline + extracted perturbed genes.

The TF-perturbation project additionally runs the model's gene calls through a Lambert + DIOPT + NCBI-synonyms master list (tf_lookup) so the verdict is a deterministic table lookup rather than a biology judgement by the LLM.

The comparator (gca compare-screen-to-gemma) joins screened output against the public Gemma REST and surfaces tag-level + gene-level agreement.

curation-proposer

For experiments already loaded into Gemma as preboardings, proposes:

• Tags (experiment-level annotations: organism part, disease model, study design)
• Factors + factor values + statements (the experimental design)
• Sample assignments (which biomaterial goes in which FV)

Each proposal is submitted as a structured Proposal to /rest/v2/datasets/{id}/curation-proposals for curator review in the UI. Default model tier is fast (haiku-4-5) — bump to standard (sonnet) or strong (opus) per request when needed.

audit

For experiments already curated in Gemma, evaluates the existing factors / FVs / tags / sample assignments against curation guidelines and emits structured AuditFinding records. Six judges:

• forbidden_efc_judge — deterministic table lookup (factor category on the forbidden list).
• fv_coverage_judge — deterministic per-FV / per-factor sample coverage checks.
• term_grounding_judge — deterministic URI-presence check on tags.
• factor_llm_judge / fv_llm_judge / tag_llm_judge — LLM judges with structured output. Each sees the audit subjects, the experiment's BioMaterial characteristics (with (constant) / (VARIES) markers), a split-dataset notice for GSEXXX.N short-names, and a silent comparison proposer's output as evidence (not ground truth). Hard rules in each prompt: missing-* findings must be anchored to the proposer's output; wrong_assignment must be grounded in evidence visible in the prompt, not in proposer-curator disagreement; per-FV evidence includes the GEO sample title so disease/condition mnemonics ground the verdict.

The auditor never rewrites curation. AuditFinding.suggested_fix is free text; materialising a fix is always a curator action in the UI (Apply & focus →). Curators triage findings (accepted / dismissed / needs_more_info / accepted+resolved) and close the audit. Dispositions on closed audits feed scripts/eval_analysis/audit_dispositions.py, which emits a dismiss-rate / parked-rate report grouped by (judge, issue_code, target_kind). See scripts/eval_analysis/README.md for output shape and the cross-repo handoff doc AUDIT_DISPOSITIONS.md in the UI repo for the full feedback-loop spec.

Mock curation API

FastAPI app under local_api/. Endpoints:

Method	Path	Purpose
GET / PUT	/datasets/{id}/design	curator-edited design (factors, FVs, statements, biomaterials, tags)
GET / PUT	/datasets/{id}/notes	per-experiment curator scratchpad
GET	/datasets/{id}/history	append-only commit log
POST	/datasets/{id}/curation-proposals	submit a proposal
GET / PATCH	/curation-proposals/{id}	curator feedback
POST / GET	/datasets/{id}/audits, /audits	submit / list audit reports
GET	/audits/{audit_id}	one audit with latest dispositions folded in
PATCH	/audits/{audit_id}	per-finding disposition (accept / dismiss / resolve)
POST	/audits/{audit_id}/finalize	close audit; gates dispositions report aggregation
POST	/audits/{audit_id}/reopen	reverse finalize
GET	/categories	canonical EFC list (28 entries, loaded from Gemma java repo)
GET	/annotations/search	typeahead + usage_count (660 entries from valueStringToOntologyTermMappings.txt)

Auth is a static bearer token from GEMMA_CURATION_API_KEY. Storage is SQLite (./curation.sqlite by default). The mock loads its category and value-mapping catalogs directly from the Gemma java repo at ~/Gemma/... (or GEMMA_RESOURCES_DIR); falls back to a small hand-curated list if the checkout isn't reachable.

the upstream Gemma REST surface tracks gaps the real Gemma REST surface needs to close (categories endpoint, usage_count on /annotations/search).

Tests

.venv/bin/python -m pytest tests/unit -q

Cross-repo collaboration

The curation UI lives in ../gemma-ui and is driven by a separate Claude instance. Coordination happens via per-feature handoff docs in that repo (AUDIT_FEATURE.md, AUDIT_DISPOSITIONS.md, PROGRESS_SSE.md). When a wire shape changes here, update the relevant handoff doc in the same commit so the UI side has a single source of truth for each feature.

Compatibility

The UI ↔ agent compatibility matrix lives in ../gemma-ui/CROSS_REPO_COMPAT.md — a single source of truth for both directions of the wire contract. Both services expose agent_version and agent_min_ui on their /health endpoint so the UI's startup CompatBanner can flag deploy-time skew.

This release: agent 0.2.1, paired with UI 0.3.0. Earlier pairings (0.2.0 ↔ 0.3.0, 0.1.0 ↔ 0.1.0) listed in the matrix.

License

Apache-2.0.