Skip to content

dev: include generated config and function docs in the local lint suite - #25525

Merged
kumarUjjawal merged 2 commits into
apache:mainfrom
kumarUjjawal:dev/21048-local-config-function-docs-check
Sep 20, 2026
Merged

kumarUjjawal merged 2 commits into
apache:mainfrom
kumarUjjawal:dev/21048-local-config-function-docs-check

Conversation

@kumarUjjawal

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

dev/rust_lint.sh does not check the generated configs.md and function pages. A contributor finds a stale page only in the config-docs-check job. That job also regenerates the tracked pages in place and then runs git diff, so the same check cannot run in a working tree with other changes.

What changes are included in this PR?

  • ci/scripts/check_config_function_docs.sh regenerates the four pages into per-run scratch directories, compares each with its working file, and prints a diff for each stale page. Check mode never rewrites a page. --write replaces the four pages, with the same clean-tree rule as rust_fmt.sh.
  • dev/update_config_docs.sh and dev/update_function_docs.sh gain --output-dir DIR, so the checker reuses the existing generators and headers. Without the option, both scripts behave as before.
  • dev/rust_lint.sh runs the checker as a write step.
  • The config-docs-check job runs the checker and keeps its git diff --exit-code guard. The job identity and setup do not change.
  • docs/source/contributor-guide/testing.md documents the check and the update command.

What is the testing strategy for this PR?

  • In a clean clone with the real generators: the no-argument scripts and --output-dir produce byte-identical pages, the four committed pages pass, a committed stale page fails with its diff, --write restores it byte for byte, the dirty-tree gate refuses, and staged edits survive check mode. No lockfile changed.
  • A fixture with stub cargo and npx covered each page stale on its own, generator and formatter failures in both modes, cleanup after failure and interruption, concurrent runs, missing npx, comparison errors, and argument errors. No page changed in any failure case.
  • The parsed workflow matches main except the replaced steps. The full ./dev/rust_lint.sh passes.

Are there any user-facing changes?

No.

🤖 Generated with Claude Code

Add `ci/scripts/check_config_function_docs.sh`, which regenerates
`docs/source/user-guide/configs.md` and the aggregate, scalar, and
window function pages and fails if a committed page differs, and run it
from `dev/rust_lint.sh` as a write step. Contributors see a stale
generated page before pushing, and `--write` replaces the four pages
under the same clean-tree rule as the other formatter scripts.

The check generates into per-run scratch directories beneath the two
documentation directories and compares directly against the working
files, so it never rewrites them in check mode, does not depend on a
repository-wide `git diff`, and leaves staged and unstaged edits and
the index alone. Scratch directories are removed on success, failure,
and interruption. A `diff` error is reported as a comparison failure,
not as stale documentation.

`dev/update_config_docs.sh` and `dev/update_function_docs.sh` gain an
`--output-dir DIR` option so the checker reuses the complete existing
generators, headers, and Prettier invocation without copying them.
Without the option both scripts behave as before, and their output is
byte-identical in either mode.

The `config-docs-check` job now runs the shared checker followed by the
existing `git diff --exit-code` guard. Its identity, condition, runner,
container, setup actions, and Node version are unchanged.

Partial progress on apache#21048.
@github-actions github-actions Bot added documentation Improvements or additions to documentation development-process Related to development process of DataFusion labels Sep 20, 2026
@codecov-commenter

codecov-commenter commented Sep 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.40%. Comparing base (b4a8c82) to head (b88d405).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25525      +/-   ##
==========================================
+ Coverage   82.38%   82.40%   +0.01%     
==========================================
  Files        1138     1138              
  Lines      434328   434951     +623     
  Branches   434328   434951     +623     
==========================================
+ Hits       357824   358406     +582     
+ Misses      54872    54844      -28     
- Partials    21632    21701      +69     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @kumarUjjawal -- this is a nice change in my mind.

I had a suggestion for some improvements but I don't think they are needed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: I suggest calling this check_generated_docs.sh as it is both shorter and more general


## Config and Function Docs Check

`docs/source/user-guide/configs.md` and the three function pages under

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also please make file references github url links so it is easier to follow what is going on?

ALso I think a bunch of details of the implementation are irrelevant for people running this check (e.g. that needs cargo and npx ,etc) is irrelevant here -- we could just say somehting like

To check they are up to date, run the ci/scripts/check_config_function_docs.sh script, which is also run as part of ./dev/rust_lint.sh

@kumarUjjawal

Copy link
Copy Markdown
Contributor Author

Thank you @alamb for suggestions.

@kumarUjjawal
kumarUjjawal added this pull request to the merge queue Sep 20, 2026
Merged via the queue into apache:main with commit 99af44a Sep 20, 2026
42 checks passed
@kumarUjjawal
kumarUjjawal deleted the dev/21048-local-config-function-docs-check branch September 20, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

development-process Related to development process of DataFusion documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants