Skip to content

Add lightweight per-tool malformed invocation recovery#280

Open
mmprotest wants to merge 1 commit into
mainfrom
mmprotest/implement-recovery-mechanism-for-malformed-invocations
Open

Add lightweight per-tool malformed invocation recovery#280
mmprotest wants to merge 1 commit into
mainfrom
mmprotest/implement-recovery-mechanism-for-malformed-invocations

Conversation

@mmprotest

Copy link
Copy Markdown
Owner

Motivation

  • Prevent model/runner loops where the model repeatedly emits malformed tool invocations for the same tool and the run spirals without useful guidance.
  • Keep the fix small, local, tool-agnostic and nondeterministic so normal exploratory behavior and existing runner control flow remain unchanged.

Description

  • Add a compact per-tool recent-failure tracker and transient degradation state on Runner using a short sliding window with _MALFORMED_TOOL_WINDOW = 4 and _MALFORMED_TOOL_THRESHOLD = 2 and lightweight escalation levels.
  • Detect only malformed invocation failures (schema/validation style errors returned by execute_tool) via _is_malformed_tool_invocation_failure, and do not count ordinary execution failures.
  • When repeated malformed invocations are observed, replace the verbose schema error payload with a very short recovery nudge produced by _recovery_message_for_tool, asking the model to re-ground before retrying while leaving the model free to choose how to recover.
  • Make the degraded state transient and inspectable: failures are recorded in a per-tool deque, degradation levels and recovery turns are tracked, and _clear_tool_degradation clears state after a valid invocation.
  • Integrate recovery with a single, surgical insertion point after tool execution (result = self._apply_malformed_tool_recovery(...)) so the runner behavior and flow are preserved without adding planners or a fallback mode.
  • Add focused unit tests that simulate sequences of tool uses and assert the recovery behaviour is triggered, cleared, and escalated correctly.

Testing

  • Added tests to tests/test_loop.py covering: test_no_recovery_message_on_single_malformed_invocation, test_recovery_triggers_after_repeated_malformed_invocations, test_degraded_state_clears_after_valid_invocation, test_immediate_relapse_after_recovery_escalates, and test_ordinary_execution_failures_do_not_count_as_malformed and supporting test helper SequencedToolClient.
  • Ran the new/modified test file with pytest -q tests/test_loop.py and the file passed: 30 passed.
  • The targeted recovery tests (the added cases) were executed and passed.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant