[RFC] Evidence-backed UI state/transition observation primitives for agent-driven behavioral reconstruction #4069

Problem / motivation

REA already provides valuable evidence-backed static and runtime analysis. For an external agent attempting large-scale behavioral reconstruction of an application it owns or is authorized to examine, the missing piece is often not another decompiler: it is a composable, evidence-backed application interaction and observation contract that can enumerate UI states and transitions, connect observations to REA evidence, and support repeatable comparisons.

This is not a request for REA to implement a one-click commercial software clone, generate replacement applications, bypass protections, or own an autonomous orchestration loop. The external agent remains responsible for exploration strategy, implementation, and iterative repair.

Requested capability / desired outcome

Please consider defining an opt-in CLI/MCP contract (or documenting existing composable equivalents) for:

1. UI state and action inventory: observable windows/pages, accessible controls, actions, navigation, modal dialogs, and state identifiers, with platform/provider and visibility limitations explicitly recorded.
2. Evidence-linked transitions: given a caller-selected action and state, return before/after observations and references to relevant process, filesystem, IPC, network, and static-analysis evidence where available; unresolved associations must remain unknown rather than guessed.
3. Repeatable scenario primitives: caller-controlled initial conditions, action sequences, bounded execution, cancellation, cleanup, and replay/comparison metadata. Do not silently promise deterministic reset of external state.
4. Coverage export: machine-readable observed states/transitions, unvisited or unreachable candidates, source evidence, and gaps, so an external agent can prioritize follow-up experiments.
5. Incremental use: stable artifact/session references and import/export suitable for long-running investigations without requiring a monolithic in-memory analysis.

A minimal first milestone could be a read-only UI state/transition evidence schema plus an adapter for one supported target class, using existing browser/process observations, rather than a universal UI automation engine.

Example workflow (proposed, not tested)

For a self-owned sample Electron app:

• An agent asks REA for an observed UI state inventory.
• It chooses a search action and supplies a bounded interaction scenario.
• REA returns before/after UI observations, available IPC/network/process evidence, and explicit unknowns.
• The agent uses those observations to implement its own equivalent feature, then compares recorded outcomes using existing comparison primitives.
• Subsequent runs reuse prior evidence and expose unexplored states without claiming exhaustive coverage.

Relationship to existing issues

I searched for related issues and do not want to duplicate them:

• [RFC/Epic] Universal Static-to-Runtime Semantic Correlation: cross-layer mapping between code, traces, packets and UI events #3718: universal static-to-runtime semantic correlation (this request would consume/extend its evidence links).
• Emit reconstruction obligation ledgers and verify coverage closure #405 and Build an evidence-backed reconstruction workspace and coverage closure workflow #268: reconstruction obligation/coverage workspace (this request focuses on collecting and exporting UI interaction state/transition evidence rather than owning closure policy).
• Add differential analysis and reconstruction verification workflows #37 and Epic: Evidence v2 and deterministic process fidelity #48: comparison and process observation foundations.
• Create a resettable macOS clean-room environment #69: resettable macOS environment (possible optional provider, not a requirement for every target).
• [Feature] Windows XAML UI reconstruction: WPF BAML, UWP/WinUI XBF, bindings, and event-handler correlation #3636: Windows XAML structural reconstruction (format-specific, complementary).

If maintainers believe existing primitives already suffice, a documented end-to-end composition example and explicit gap inventory would also address much of this request.

Scope / safety / verification

• Authorized, self-owned or suitably licensed sample applications only.
• No anti-debugging/DRM/license bypass, stealth instrumentation, credential extraction, or attack automation.
• Any execution, input injection, filesystem/network mutation, or state reset must declare its authority, side effects, time/resource limits, and cleanup; request explicit maintainer scope review before adding a provider with new execution powers.
• Validate against a small self-owned sample with known UI transitions and intentionally missing/ambiguous observations. Check that unknowns remain unknown, that cancellation/cleanup work, and that output is stable enough for external agents.
• No runtime implementation or validation has been performed for this proposal.

Questions for maintainers

1. Is a provider-neutral UI interaction/observation evidence contract within REA's intended infrastructure scope?
2. Should this be tracked under [RFC/Epic] Universal Static-to-Runtime Semantic Correlation: cross-layer mapping between code, traces, packets and UI events #3718 / Emit reconstruction obligation ledgers and verify coverage closure #405 instead of as a separate feature?
3. Which initial platform would best fit existing providers (browser, Electron, or macOS native)?