Skip to content

About

Job-market intelligence pipeline that turns raw job postings into structured, queryable skill-demand signals.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

6 Commits

Folders and files

Repository files navigation

RoleSignal

Job-market intelligence from raw postings to queryable skill-demand signals.

Python · PostgreSQL · Pydantic · SQLAlchemy · LLM Structured Outputs

Overview

RoleSignal turns public job postings into structured, queryable signals about technical skill demand. It uses a hybrid deterministic/LLM classifier to identify relevant engineering roles, extracts structured requirements, normalizes technical competencies, and persists the resulting data in PostgreSQL for analytics.

Its central design is deterministic-first: a conservative title router handles high-confidence cases, while semantic classification runs only when the router explicitly abstains. The LLM is therefore a narrow, typed component inside an auditable data pipeline—not the pipeline itself.

Benchmark snapshot

Seven-class accuracy Macro-F1 Target-vs-other F1 Deterministic coverage
89.6% 87.5% 98.8% 56.25%

These results come from a frozen, prediction-blind benchmark of 48 real job postings across six Greenhouse boards. The benchmark is small and imbalanced, and its labels are model-generated silver labels rather than independently human-gold ground truth. Target precision was 100% and target recall was 97.6%.

Deterministic routing avoided semantic fallback for 27 of 48 benchmark examples. This is a routing call-avoidance count, not a monetary-savings claim. See the frozen benchmark summary for methodology, provenance, detailed results, and limitations.

How it works

flowchart TD
    A[Greenhouse ingestion] --> B[Validated RawJob]
    B --> C[Deterministic title router]
    C -->|Resolved| E[Role family]
    C -->|Explicit abstention| D[Semantic fallback]
    D --> E
    E --> F{Target role?}
    F -->|No| G[Retain posting<br/>Skip extraction]
    F -->|Yes| H[Requirement extraction]
    H --> I[Pydantic validation]
    I --> J[Deterministic competency normalization]
    J --> K[(PostgreSQL)]
    K --> L[Prevalence, category, and<br/>co-occurrence analytics]
Loading

Non-target postings remain auditable, failures are isolated per posting, and typed boundaries keep ingestion, classification, extraction, normalization, and persistence separate.

Engineering highlights

  • Deterministic-first classification: high-confidence titles resolve locally; ambiguous cases produce an explicit abstention before semantic fallback.
  • Typed model boundary: provider output is schema-constrained and validated with Pydantic before it enters the pipeline.
  • Conservative normalization: deterministic aliases preserve unknown and partially resolved mentions instead of hiding them behind fuzzy matching.
  • Auditable lifecycle: per-job state and persistence retain rejected work while isolating individual posting failures.

Quick start

Requirements:

  • Python 3.12+
  • uv
  • PostgreSQL

Install dependencies, create the local environment file, apply migrations, and run the offline test suite:

uv sync --dev
Copy-Item .env.example .env
uv run --env-file .env alembic upgrade head
uv run pytest

Configure DATABASE_URL and the selected provider credentials in .env. Then inspect a deliberately small Greenhouse batch (Gemini is the default provider):

uv run --env-file .env python scripts/inspect_greenhouse_batch.py remotecom --limit 10

Query skill prevalence, category, and co-occurrence analytics after processing jobs:

uv run --env-file .env python scripts/show_skill_analytics.py `
  --category database --anchor Python

Run the deterministic normalization diagnostic without network access:

uv run python scripts/evaluate_skill_normalization.py

Evaluation

Classification was evaluated on a frozen 48-posting benchmark spanning six Greenhouse boards. The repository includes methodology, schemas, evaluation tooling, annotation guidance, and a description-free public result summary. The public benchmark summary is the primary result artifact and contains provenance, detailed performance and routing results, and limitations. Raw description-bearing evaluation artifacts are intentionally excluded from the public repository; detailed reproducibility information remains in the role-classification evaluation guide.

Limitations

  • The 48-example benchmark spans six Greenhouse boards, has uneven class support, and uses silver rather than independently human-gold labels.
  • Classification metrics do not establish requirement-extraction quality, end-to-end production reliability, or generalization.
  • Normalization is registry-based and deliberately leaves unsupported mentions unresolved.

About

Job-market intelligence pipeline that turns raw job postings into structured, queryable skill-demand signals.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages