How are you guys stopping agents from silent loops/stampedes? Standard step_counters feel like garbage

[Minakshi Aggarwal (minakshihub)]

I’ve been looking at multi-agent failure states, and it seems like everyone is relying on basic Python step_counters or LLM prompt-engineering ("do not repeat yourself") to prevent agents from falling into infinite tool-calling loops or stampeding APIs.

Coming from a low-level systems/C++ background, handling loop-prevention at the application layer feels insane. It adds latency, and when a timeout triggers, it just leaves the state half-executed.

Has anyone tried intercepting tool calls deterministically via a dedicated proxy or trajectory ledger? E.g., a lightweight interceptor that hard-kills the process the exact microsecond an identical state or argument repeats?

Curious what the actual production solution is, or if everyone is just accepting the latency and hoping the prompt holds?

[Rafay M. (rafayaar)]

Minakshi Aggarwal (@minakshihub)

I’d probably handle this at the tool boundary rather than relying only on prompts or max-step limits.

One approach is to keep a small per-run history of (tool, args, state) and block a call only when the same action keeps repeating without any state change. That avoids killing legitimate retries or polling.

In AutoGen, an InterventionHandler looks like a good place to implement this before the tool actually runs.

I’d still keep timeouts / max iterations as a final safety net, but use the trajectory check for the actual loop detection.

[Minakshi Aggarwal (minakshihub)]

Appreciate the pragmatic take, Rafay. The logic you described—tracking a trajectory history of (tool, args, state) and blocking redundant hits—is exactly the right mechanism.

My concern with pushing that into an InterventionHandler (or any Python-layer interceptor) is the compute bottleneck at scale. When you have dozens of concurrent agents, hashing large state objects and traversing that history inside Python's single event loop chokes the CPU and spikes latency across the whole pod.

This is actually why I’m architecting a way to pull your exact logic down into an out-of-process C++ proxy (https://github.com/minakshihub/aegis). It tracks that exact trajectory history and drops the socket on a repeating hash, but it does it at bare-metal speeds without blocking the Python runtime.

Are you seeing the Python event loop hold up well when calculating those trajectory histories under heavy concurrent load?

[Rafay M. (rafayaar)]

Minakshi Aggarwal (@minakshihub)

Yeah, that’s a fair point. I probably wouldn’t hash the full state every time either. I was thinking more of a lightweight fingerprint or some small state marker.
Moving that logic outside Python makes sense if you’re actually seeing CPU pressure at scale.
The part I’d be curious about with Aegis is how you distinguish a real loop from a legit repeated call, like polling or retries. If the request looks the same but the context is different, I guess that’s where it gets tricky.

[Ama Senevirathne (amasen02)]

You're completely right to be frustrated by prompt-level "do not repeat yourself" or crude step counters. Under real production loads, prompt guards degrade as context grows, and raw step counters just let an agent run up your LLM bill before violently terminating mid-transaction, leaving orphaned locks and corrupted state.

Hard-killing the worker process on the spot has the same problem in reverse: you drop the connection, leak external resources, and leave the caller with a 500 error instead of a graceful degradation.

What works reliably in production is a deterministic circuit breaker at the dispatch/interception layer. Here is the architecture we use to handle both loops and stampedes:

1. Sliding Window Trajectory Ledger (Cycle Detection)

Instead of inspecting full LLM responses, calculate a canonical hash of the tool invocation:
fingerprint = sha256(f"{tool_name}:{json.dumps(args, sort_keys=True)}")

Maintain a ring buffer of the last $N$ calls (typically $k=5$). If the identical fingerprint appears 2–3 times in the window—especially if the intermediate tool output returned an error or an empty diff—the interceptor trips immediately before executing the tool.

Crucially: do not kill the process. Inject a synthetic observation directly into the agent's scratchpad:

{
  "status": "circuit_breaker_tripped",
  "error": "DeterministicLoopDetected: Tool 'query_database' invoked with identical parameters 3 times without state progress. Branch aborted. Summarize what you have or request operator intervention."
}

This forces the model to synthesize a final response or trigger a fallback handler without wasting tokens or deadlocking.

2. Concrete Interceptor Hook

In AutoGen (or any agent runtime with a pre-tool execution hook), you wrap the tool dispatcher:

import hashlib
import json
from collections import deque

class TrajectoryCircuitBreaker:
    def __init__(self, max_consecutive_duplicates: int = 2, window_size: int = 6):
        self.max_dups = max_consecutive_duplicates
        self.history = deque(maxlen=window_size)

    def _fingerprint(self, tool_name: str, args: dict) -> str:
        serialized = json.dumps(args, sort_keys=True)
        return hashlib.sha256(f"{tool_name}:{serialized}".encode()).hexdigest()

    def check_and_record(self, tool_name: str, args: dict) -> bool:
        fp = self._fingerprint(tool_name, args)
        dup_count = sum(1 for item in self.history if item == fp)
        self.history.append(fp)
        # Trip if identical invocation threshold exceeded
        return dup_count < self.max_dups

# In your tool hook / middleware:
breaker = TrajectoryCircuitBreaker(max_consecutive_duplicates=2)

def tool_execution_interceptor(tool_name: str, args: dict):
    if not breaker.check_and_record(tool_name, args):
        return {
            "error": f"CircuitBreakerTripped: Repeated calls to '{tool_name}' detected. Tool execution halted."
        }
    return execute_actual_tool(tool_name, args)

3. API Stampede Protection at the Gateway

For stampedes across multiple agents or swarm workers, never trust agents to coordinate rate limits. Enforce it at the tool gateway using:

• Token Bucket / Leaky Bucket (e.g. Redis rate-limiter or a local asyncio.Semaphore per external tool domain).
• Idempotency Keys: Compute a hash of (run_id, tool_name, args) so duplicate tool triggers within a 5-second window return the cached pending promise instead of hitting the upstream service twice.

This keeps all execution deterministic at the infrastructure boundary and completely decouples stability from whatever the LLM feels like outputting.

[solim (soul-sol)]

The part that helped me was separating two things that step counters conflate:
bounding the run and deciding what happened when it ends.

Bounding I moved out of the agent entirely — a hard external timeout around the
process, so nothing inside the loop has to be trusted to stop itself:

perl -e 'alarm shift; exec @ARGV' 1800 <agent-command> < /dev/null > run.log 2>&1

exec replaces the process, so the child's exit status passes through unchanged, and
the alarm always fires. That's the low-level guarantee the application layer can't give
you.

Deciding what happened is the part that quietly went wrong for me. Two rules, both
learned the hard way:

• Record the exit status out of band. A background process's status disappears the
  moment the parent stops waiting, so the launcher writes $? to a file and the watcher
  reads that file — never the log tail.
• Never treat a missing completion marker as a stall. One tool prints a token-usage
  line on normal exit; others print nothing. My watcher required that marker from every
  tool and reported perfectly healthy runs as stuck. Exit code plus reading the result
  body is the only judgement that survived — and an exit of 0 can still mean "the agent
  only asked a question", so a clean exit is a candidate, not a verdict.

On stampedes specifically: a timeout that kills without recording the status is worse
than no timeout, because the caller now has no information and usually retries. The
timeout and the exit-status record have to ship together.

Scripts, MIT: https://github.com/soul-sol/agent-watch

[Minakshi Aggarwal (minakshihub)]

Ama Senevirathne (@amasen02) The sliding window ledger is definitely the right mental model for loop detection. However, injecting a synthetic error back into the LLM’s scratchpad is exactly the pattern I am trying to escape. Under heavy load, if an agent is stuck in a "Cognitive Stall" (endless apologies) or a "Read-Only Spiral," feeding it an error prompt just burns more tokens as it apologizes and loops again. We can't rely on the LLM to govern itself when its context window is already degraded.

Rafay M. (@rafayaar) To answer your question on how Aegis handles legitimate polling vs. loops: Aegis doesn't just hash the tool and arguments; it tracks the returning state code and enforces streak limits. If an agent polls a database 5 times and the data changes, the streak resets. It only triggers the hardware limit if it hits the exact same hash with no state progression (Action Thrashing).

solim (@soul-sol) Hard timeouts are definitely a necessary safety net, but waiting for an alarm to fire while an agent is caught in a high-speed token hemorrhage is a massive billing risk.

The architecture I just locked down for Aegis handles this by returning a 16-bit deterministic dictionary flag (e.g., Flag 256 for Slow Token Boil, Flag 512 for Read-Only Spirals) in O(1) time directly to the Python adapter. The Python layer acts as the traffic cop—it reads the flag and can cleanly sever the LLM worker instantly, without dropping the main application process or negotiating with the LLM.

[Anatolii Maltsev (maltsev-dev)]

step_counter is the wrong layer for this - it's prompt-side, it gets bypassed by re-prompting or by code-generated tool calls that don't go through the agent loop. We tried it, it doesn't hold.

The thing that finally worked was intercepting tool calls at the runtime boundary: every call gets a server-minted execution_id and is checked against a budget before dispatch. If the budget is blown, the gateway returns block (no exception, no kill) and the agent can choose to summarize and return.
The crucial trick is binding the check to the action payload, not the conversation - so a fresh agent thread can't reset it.
Latency is ~1ms because the check is a Redis read, not an LLM call.
Also worth wiring half-open circuit semantics into the spend cap so a transient 429 doesn't disable a provider for 24h.

[Ama Senevirathne (amasen02)]

Minakshi Aggarwal (@minakshihub) That distinction is spot-on. You're completely right about the "Cognitive Stall" failure mode: once an agent's attention window begins degraded looping, feeding an error back into the scratchpad just triggers the classic meta-apology spiral ("I apologize for the confusion, let me try a different approach...") while burning expensive output tokens and remaining stuck in the loop. Asking the LLM to police its own broken context is fundamentally flawed.

Severing at the deterministic Python harness layer via the 16-bit status flag without LLM negotiation is much cleaner architecture:

1. Deterministic Cutoff: The Python runtime acts as the strict supervisor. When the streak threshold on duplicate action hashes with static environment state is reached, the harness halts the worker process immediately (O(1) abort).
2. Zero-Token Waste: No recursive completion generation or self-reflection prompts.
3. Clean Teardown: The caller receives an actionable, structured error code (e.g., Action Thrashing / Read-Only Spiral) rather than a drained token budget or a timeout stacktrace.

Enforcing state progression rather than just hashing inputs/outputs handles real-world polling cleanly too. Appreciate you sharing the Aegis architecture pattern here!

[Minakshi Aggarwal (minakshihub)]

Ama, thank you. You captured the exact physics of the failure mode perfectly—the "meta-apology spiral" is the precise term for what happens when we ask a model to police its own broken state.

Because you see the necessity of this deterministic boundary so clearly, I wanted to share how we are putting this into practice. We recently built Phase 1 of Aegis specifically as a passive observer (a zero-latency diagnostic wiretap).

Before a system can cleanly tear down a cognitive stall, it needs the forensic data to know why the loop started. The observer sits entirely out-of-process, hashes the trajectories, and captures the exact chronological steps that caused the anomaly without altering the application's performance. It gives engineering teams the raw "flight data" needed to actually diagnose the pitfall and fix their models' boundaries, rather than just blindly terminating the crash.

Thrilled to have this architectural alignment with you. Thanks for the brilliant breakdown!

[danpingsmile1197-design]

Just wanted to clarify something from my earlier comment about jzstoken pricing. The "$15 lasts 2 weeks" number is super rough and depends heavily on which models you're using and how many agent loops you're running.

What's actually consistent: jzstoken is genuinely one of the cheapest options for agent workloads. Their own model jzs-max-3.0 is only $0.062/1M input tokens. That's pennies per request even for long agent loops.

47 models under one key, pay-per-token, no subscription. $1.50 to start, credits never expire.

For agent workloads, using cheap models for routine tasks and expensive ones only when needed is a game-changer for costs.

[zhufeng (fujiezee)]

One part of the original question hasn't been addressed yet: when a loop breaker fires, the run is left half-executed. Whichever mechanism trips (fingerprint window, hard timeout, out-of-process proxy), it helps to make "what was done" computable at that moment, so tripping isn't the end of the story.

A small pattern that tends to work with any of them:

1. Before each side-effecting tool call, write an intent record (tool, args digest, idempotency key). After the call, write the outcome. Reads don't need it.
2. When a breaker trips, the run ends in an explicit state such as stopped-unverified rather than failed or done, and the report lists intents with no outcome, plus outcomes that have no compensation.
3. Each tool declares its safe abort once, up front: nothing to undo, a named compensation, or needs a person. Anything irreversible defaults to needs a person.
4. A retry or resume reads the ledger first, so a repeated call with the same idempotency key is skipped instead of re-run.

That also answers the polling vs loop question from a different angle: a poll is a read, so it never enters the ledger, while a repeated side effect with the same key is a stampede by definition.

[Minakshi Aggarwal (minakshihub)]

Zhufeng, this is exactly the right architectural pattern. Implementing an intent-based Write-Ahead Log (intent → side-effect → outcome) at the tool-call boundary is the only mathematically sound way to recover a stopped-unverified state.

However, there is a brutal physical bottleneck when executing this at scale: writing that intent record synchronously before every side-effecting call introduces a massive I/O serialization tax if you are logging it to a standard database or disk.

This is exactly why I designed the out-of-process proxy as a C++17 shared-memory ring buffer.

Instead of writing the intent record to a slow native log, the agent pushes the (tool, args digest, idempotency key) struct directly into the lock-free ring buffer in shared memory. The latency cost drops to nanoseconds—zero disk I/O, zero syscalls on the happy path.

When the loop breaker actually trips, the proxy dumps those rolling execution cycles to disk. That core dump becomes the exact "unverified ledger" you described—showing the intent with no outcome—but you didn't have to pay the synchronous latency tax to record it.

Are you currently pushing these intent records to an in-memory store like Redis to avoid the I/O drag, or are you logging them straight to a native file/DB?