Skip to content

Validate Jobflow workflow outputs - #3405

Open
Andrew-S-Rosen wants to merge 26 commits into
mainfrom
agent/strengthen-jobflow-results
Open

Validate Jobflow workflow outputs#3405
Andrew-S-Rosen wants to merge 26 commits into
mainfrom
agent/strengthen-jobflow-results

Conversation

@Andrew-S-Rosen

@Andrew-S-Rosen Andrew-S-Rosen commented Aug 10, 2026

Copy link
Copy Markdown
Member

Summary

  • Make every tests/jobflow execution inspect a concrete returned result.
  • Resolve upstream quacc jobs automatically inside Jobflow job arguments and ordinary containers.
  • Remove explicit .output composition from quacc-decorated Jobflow workflows.
  • Preserve native Jobflow syntax in tests that intentionally demonstrate @jf.job.
  • Use the plain file-transfer specification API introduced by Replace Copy with plain file-transfer specs #3409 for deferred copy_files directory references.
  • Remove the obsolete Jobflow-specific JobflowCopy compatibility wrapper after merging current main.

Why

ensure_success=True only confirms that Jobflow did not mark execution as failed. It does not prove that the expected value was returned or that downstream jobs received usable outputs. Quacc workflows also need to remain workflow-engine agnostic rather than requiring Jobflow-specific .output access.

PR #3409 replaced custom copy wrappers with ordinary list-of-dictionary transfer specifications. Keeping the old Jobflow wrapper after merging that PR caused an import from the deleted job_argument.py module, so this branch now relies entirely on Jobflow's normal argument traversal.

Validation

  • Jobflow test suite: 27 passed
  • Ruff lint and formatting checks
  • Final audit: all jf.run_locally calls in tests/jobflow capture and inspect responses
  • Final audit: no explicit .output calls in src/quacc/recipes

@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.48%. Comparing base (842151c) to head (f9b3f2a).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3405   +/-   ##
=======================================
  Coverage   99.48%   99.48%           
=======================================
  Files          99       99           
  Lines        4258     4258           
=======================================
  Hits         4236     4236           
  Misses         22       22           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vineetbansal ChatGPT had to make several changes to the source code. I imagine this may not be necessary and that there is some simpler change we can be making upstream in jobflow to avoid this.

@Andrew-S-Rosen
Andrew-S-Rosen marked this pull request as ready for review August 11, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant