feat(dq): add dq_run_job behind a data-quality-experimental flag (DEV-211807) - #140
Draft
bafuchs-collibra wants to merge 1 commit into
Draft
feat(dq): add dq_run_job behind a data-quality-experimental flag (DEV-211807)#140bafuchs-collibra wants to merge 1 commit into
bafuchs-collibra wants to merge 1 commit into
Conversation
Trigger a run of an existing data-quality job as-is, wrapping the public
POST /rest/dq/1.0/jobs/{jobName}/run endpoint. The job's definition is
never modified — this only queues execution.
Built around a confirm checkpoint, per TOOL_CONTRIBUTION_STANDARDS.md §5:
queueing a run writes (it consumes compute on the job's edge site, and a
backrun can queue many runs at once), so confirm=false (the default)
resolves the job and returns a runPlan echoing every field that would be
submitted, queueing nothing; confirm=true performs the run. The runPlan
covers the resolved job name, run-date window and backfill so the user
approves exactly what gets sent (§5.2).
Job names resolve the same way as dq_get_job: exact match first, falling
back to a fuzzy name search on 404 — no matches errors, one match is used,
several are returned as candidates for the caller to disambiguate. A
typo'd name therefore reports clearly instead of failing ambiguously.
Run parameters are all optional; omitting them runs the job with its
current settings, matching normal scheduled behavior. runDate/runDateEnd
override the time slice and backrun queues prior periods. Pairs are
validated before any network call, so a half-supplied override never
surfaces as a downstream 400.
Registered behind a new `data-quality-experimental` feature flag, which
gates data-quality tools that are still experimental. This is separate
from the generally-available data-quality tools, which register
unconditionally since #126 removed the old data-quality flag: a tool
graduates by moving out of the gated block rather than by changing what
the flag means. Listed in the README and --experimental help text
alongside its siblings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bafuchs-collibra
force-pushed
the
feat/DEV-211807
branch
from
September 10, 2026 19:58
b57daaa to
aa7e8a1
Compare
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 What does this PR do?
Adds
dq_run_job, a tool that triggers a run of an existing Collibra data quality jobas-is via the public
POST /rest/dq/1.0/jobs/{jobName}/run. The job's definition is nevermodified — this only queues execution.
TOOL_CONTRIBUTION_STANDARDS.md§5). Queueing a run writes: itconsumes compute on the job's edge site, and a backrun can queue many runs at once. So
confirm=false(the default) is read-only — it resolves the job and returns arunPlanechoing every field that would be submitted, queueing nothing.
confirm=trueperforms the run.dq_get_job: exact match first, falling back to a namesearch on 404. No matches errors clearly, one match is used, several are returned as candidates
to disambiguate.
runDate/runDateEndoverride the time slice;backrunqueues prior periods. Pairs are validated beforeany network call, so a half-supplied override never surfaces as a downstream 400.
Feature flag — please read, this is the part worth reviewing
dq_run_jobis gated behind a newdata-quality-experimentalexperimental flag, not the olddata-qualityone, which #126 removed.This implements the two-flag split discussed with @philrhinehart-collibra: a GA data quality
flag for shipped tools, and a separate experimental flag for preview tools. Per that
discussion, this flag is "only for these new tools which are experimental" — it deliberately
does not re-gate the 19 generally-available DQ tools, which continue to register
unconditionally. Re-gating those is the GA-flag work and is out of scope here.
A tool graduates by moving out of the gated block, rather than by changing what a flag means.
Related context: #126 removed the
data-qualityflag; #129 (revert) was closed unmerged.Testing
Tested on https://udq.dev-aws.cp.collibra-ops.com/ — preview, confirmed run, and fuzzy job-name
resolution verified end to end (run
f12298a6-3dd4-4910-8322-c5c698637f9c, confirmed viadq_get_job_run).Rebased onto
main(cfe51c9):go build ./...clean,go vet ./...clean,gofmtclean,52/52 test packages pass, 13/13 tests in the new package.
Open questions for reviewers
These are deliberate decisions left to maintainers, not oversights.
run_data_quality_job),but the sibling tools all use the
dq_*prefix — the family is now eight deep. Consistencyargues for keeping
dq_run_job, but maintainers should rule on the whole family rather thanthis one tool.
data-quality-experimentalthe right name for the experimental half, and should theGA flag reuse
data-quality? Worth settling alongside the GA flag work so the two names landtogether.
the API contract changes. That mechanism ships in
.09; nothing exists yet, so none isincluded here.
main's README still marks eight now-GA DQ tools as"Experimental (
data-qualityfeature flag)" — left untouched here to keep this diff to oneREADME line, but it should be cleaned up. (b) The Chip Tool Personas and Permission Mapping
page lists this tool under the wrong PR (feat(dq): add DQ rule lifecycle + job observability MCP tools #91), and lists
get_dq_job_status/get_dq_job_logfor what actually shipped asdq_get_job_runin DEV-205658: Add dq_get_job and dq_get_job_run tools #122. That's @EricWarnerCollibra's page — flagging, not editing.Jira
DEV-211807
✅ Checklist
🤖 Generated with Claude Code