feat(evaluation) 11/15: admin API routes, and promptfoo in both images

[Ahmath-Gadji]

Part 11 of 15 of the split of #811. Targets eval/10-ray-worker (#821). Completes the backend — the feature is fully usable over HTTP after this; parts 12–14 are the UI.

What

Method	Path
GET	/evaluation/datasets	list
POST	/evaluation/datasets	upload corpus + test set
DELETE	/evaluation/datasets/{id}	delete
GET	/evaluation/runs	history
POST	/evaluation/runs	queue a run (202, or 409 / 503)
GET	/evaluation/runs/{id}	one run with metrics and per-question detail
POST	/evaluation/runs/{id}/cancel	abandon an in-flight run

Plus the response schemas, and Node 22 + a pinned promptfoo in both images.

Notable

• Every route is admin-only, via a router-level Depends(require_admin). A run indexes a corpus, spends grader tokens and occupies the single runner slot — not something a partition editor should be able to trigger.
• The router passes streams, not bytes. create_dataset hands UploadFile.file straight to the service. Starlette has already spooled a large upload to disk; reading it here would pull it back into memory unbounded and undo part 7's streaming.
• promptfoo is installed in api.Dockerfile and ray.Dockerfile. Compose runs Ray inside the API container, so the API image needs it; a deployment with a separate Ray cluster needs the ray image. Missing either produces a run that fails only at the promptfoo step, minutes in.
• Pinned, not npx promptfoo@latest. A run should not depend on npm reachability, nor on CLI behaviour changing under a deployment that was never rebuilt. Node comes from NodeSource because the distro package predates promptfoo's floor.

Testing

ruff, format check, the layer-import guard and the full unit suite (2270 passed; the one failure reproduces on develop). The routes are thin adapters over the service tested in parts 7 and 9.

[coderabbitai]

Warning

Review limit reached

@EnjoyBacon7, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 59 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 54b314f0-dffb-4e09-899d-9dc122bf1e15

📥 Commits

Reviewing files that changed from the base of the PR and between 6497847 and a229934.

📒 Files selected for processing (5)

• infra/docker/api.Dockerfile
• infra/docker/ray.Dockerfile
• openrag/api/main.py
• openrag/api/routers/admin/evaluation.py
• openrag/api/schemas/admin/evaluation_schemas.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch eval/11-api-routes

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Ahmath-Gadji]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 56 minutes.