Skip to content

feat(dq): add dq_run_job behind a data-quality-experimental flag (DEV-211807) - #140

Draft
bafuchs-collibra wants to merge 1 commit into
mainfrom
feat/DEV-211807
Draft

feat(dq): add dq_run_job behind a data-quality-experimental flag (DEV-211807)#140
bafuchs-collibra wants to merge 1 commit into
mainfrom
feat/DEV-211807

Conversation

@bafuchs-collibra

@bafuchs-collibra bafuchs-collibra commented Sep 10, 2026

Copy link
Copy Markdown

🎯 What does this PR do?

Adds dq_run_job, a tool that triggers a run of an existing Collibra data quality job
as-is via the public POST /rest/dq/1.0/jobs/{jobName}/run. The job's definition is never
modified — this only queues execution.

  • Confirm checkpoint (TOOL_CONTRIBUTION_STANDARDS.md §5). Queueing a run writes: it
    consumes compute on the job's edge site, and a backrun can queue many runs at once. So
    confirm=false (the default) is read-only — it resolves the job and returns a runPlan
    echoing every field that would be submitted, queueing nothing. confirm=true performs the run.
  • Fuzzy job-name resolution, matching dq_get_job: exact match first, falling back to a name
    search on 404. No matches errors clearly, one match is used, several are returned as candidates
    to disambiguate.
  • Optional run parameters. Omitting them runs the job with its current settings. runDate /
    runDateEnd override the time slice; backrun queues prior periods. Pairs are validated before
    any network call, so a half-supplied override never surfaces as a downstream 400.

Feature flag — please read, this is the part worth reviewing

dq_run_job is gated behind a new data-quality-experimental experimental flag, not the old
data-quality one, which #126 removed.

This implements the two-flag split discussed with @philrhinehart-collibra: a GA data quality
flag
for shipped tools, and a separate experimental flag for preview tools. Per that
discussion, this flag is "only for these new tools which are experimental" — it deliberately
does not re-gate the 19 generally-available DQ tools, which continue to register
unconditionally. Re-gating those is the GA-flag work and is out of scope here.

A tool graduates by moving out of the gated block, rather than by changing what a flag means.

Related context: #126 removed the data-quality flag; #129 (revert) was closed unmerged.

Testing

Tested on https://udq.dev-aws.cp.collibra-ops.com/ — preview, confirmed run, and fuzzy job-name
resolution verified end to end (run f12298a6-3dd4-4910-8322-c5c698637f9c, confirmed via
dq_get_job_run).

Rebased onto main (cfe51c9): go build ./... clean, go vet ./... clean, gofmt clean,
52/52 test packages pass, 13/13 tests in the new package.

Open questions for reviewers

These are deliberate decisions left to maintainers, not oversights.

  1. Naming (§4.1). The standards doc says to spell out abbreviations (run_data_quality_job),
    but the sibling tools all use the dq_* prefix — the family is now eight deep. Consistency
    argues for keeping dq_run_job, but maintainers should rule on the whole family rather than
    this one tool.
  2. Gating. Is data-quality-experimental the right name for the experimental half, and should the
    GA flag reuse data-quality? Worth settling alongside the GA flag work so the two names land
    together.
  3. Contract test (§8.3). The standards doc anticipates a DQ-side contract test that fails if
    the API contract changes. That mechanism ships in .09; nothing exists yet, so none is
    included here.
  4. Stale docs, not fixed in this PR. (a) main's README still marks eight now-GA DQ tools as
    "Experimental (data-quality feature flag)" — left untouched here to keep this diff to one
    README line, but it should be cleaned up. (b) The Chip Tool Personas and Permission Mapping
    page lists this tool under the wrong PR (feat(dq): add DQ rule lifecycle + job observability MCP tools #91), and lists get_dq_job_status /
    get_dq_job_log for what actually shipped as dq_get_job_run in DEV-205658: Add dq_get_job and dq_get_job_run tools #122. That's @EricWarnerCollibra's page — flagging, not editing.

Jira

DEV-211807

✅ Checklist

  • My code follows the style guidelines of this project.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation (if needed).
  • My commit messages follow the Conventional Commits standard.

🤖 Generated with Claude Code

Trigger a run of an existing data-quality job as-is, wrapping the public
POST /rest/dq/1.0/jobs/{jobName}/run endpoint. The job's definition is
never modified — this only queues execution.

Built around a confirm checkpoint, per TOOL_CONTRIBUTION_STANDARDS.md §5:
queueing a run writes (it consumes compute on the job's edge site, and a
backrun can queue many runs at once), so confirm=false (the default)
resolves the job and returns a runPlan echoing every field that would be
submitted, queueing nothing; confirm=true performs the run. The runPlan
covers the resolved job name, run-date window and backfill so the user
approves exactly what gets sent (§5.2).

Job names resolve the same way as dq_get_job: exact match first, falling
back to a fuzzy name search on 404 — no matches errors, one match is used,
several are returned as candidates for the caller to disambiguate. A
typo'd name therefore reports clearly instead of failing ambiguously.

Run parameters are all optional; omitting them runs the job with its
current settings, matching normal scheduled behavior. runDate/runDateEnd
override the time slice and backrun queues prior periods. Pairs are
validated before any network call, so a half-supplied override never
surfaces as a downstream 400.

Registered behind a new `data-quality-experimental` feature flag, which
gates data-quality tools that are still experimental. This is separate
from the generally-available data-quality tools, which register
unconditionally since #126 removed the old data-quality flag: a tool
graduates by moving out of the gated block rather than by changing what
the flag means. Listed in the README and --experimental help text
alongside its siblings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bafuchs-collibra bafuchs-collibra changed the title feat(dq): add dq_run_job behind a data-quality-preview flag (DEV-211807) feat(dq): add dq_run_job behind a data-quality-experimental flag (DEV-211807) Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant