Job-market intelligence from raw postings to queryable skill-demand signals.
Python · PostgreSQL · Pydantic · SQLAlchemy · LLM Structured Outputs
RoleSignal turns public job postings into structured, queryable signals about technical skill demand. It uses a hybrid deterministic/LLM classifier to identify relevant engineering roles, extracts structured requirements, normalizes technical competencies, and persists the resulting data in PostgreSQL for analytics.
Its central design is deterministic-first: a conservative title router handles high-confidence cases, while semantic classification runs only when the router explicitly abstains. The LLM is therefore a narrow, typed component inside an auditable data pipeline—not the pipeline itself.
| Seven-class accuracy | Macro-F1 | Target-vs-other F1 | Deterministic coverage |
|---|---|---|---|
| 89.6% | 87.5% | 98.8% | 56.25% |
These results come from a frozen, prediction-blind benchmark of 48 real job postings across six Greenhouse boards. The benchmark is small and imbalanced, and its labels are model-generated silver labels rather than independently human-gold ground truth. Target precision was 100% and target recall was 97.6%.
Deterministic routing avoided semantic fallback for 27 of 48 benchmark examples. This is a routing call-avoidance count, not a monetary-savings claim. See the frozen benchmark summary for methodology, provenance, detailed results, and limitations.
flowchart TD
A[Greenhouse ingestion] --> B[Validated RawJob]
B --> C[Deterministic title router]
C -->|Resolved| E[Role family]
C -->|Explicit abstention| D[Semantic fallback]
D --> E
E --> F{Target role?}
F -->|No| G[Retain posting<br/>Skip extraction]
F -->|Yes| H[Requirement extraction]
H --> I[Pydantic validation]
I --> J[Deterministic competency normalization]
J --> K[(PostgreSQL)]
K --> L[Prevalence, category, and<br/>co-occurrence analytics]
Non-target postings remain auditable, failures are isolated per posting, and typed boundaries keep ingestion, classification, extraction, normalization, and persistence separate.
- Deterministic-first classification: high-confidence titles resolve locally; ambiguous cases produce an explicit abstention before semantic fallback.
- Typed model boundary: provider output is schema-constrained and validated with Pydantic before it enters the pipeline.
- Conservative normalization: deterministic aliases preserve unknown and partially resolved mentions instead of hiding them behind fuzzy matching.
- Auditable lifecycle: per-job state and persistence retain rejected work while isolating individual posting failures.
Requirements:
- Python 3.12+
uv- PostgreSQL
Install dependencies, create the local environment file, apply migrations, and run the offline test suite:
uv sync --dev
Copy-Item .env.example .env
uv run --env-file .env alembic upgrade head
uv run pytestConfigure DATABASE_URL and the selected provider credentials in .env. Then
inspect a deliberately small Greenhouse batch (Gemini is the default provider):
uv run --env-file .env python scripts/inspect_greenhouse_batch.py remotecom --limit 10Query skill prevalence, category, and co-occurrence analytics after processing jobs:
uv run --env-file .env python scripts/show_skill_analytics.py `
--category database --anchor PythonRun the deterministic normalization diagnostic without network access:
uv run python scripts/evaluate_skill_normalization.pyClassification was evaluated on a frozen 48-posting benchmark spanning six Greenhouse boards. The repository includes methodology, schemas, evaluation tooling, annotation guidance, and a description-free public result summary. The public benchmark summary is the primary result artifact and contains provenance, detailed performance and routing results, and limitations. Raw description-bearing evaluation artifacts are intentionally excluded from the public repository; detailed reproducibility information remains in the role-classification evaluation guide.
- The 48-example benchmark spans six Greenhouse boards, has uneven class support, and uses silver rather than independently human-gold labels.
- Classification metrics do not establish requirement-extraction quality, end-to-end production reliability, or generalization.
- Normalization is registry-based and deliberately leaves unsupported mentions unresolved.