ARCHITECTURE.md

Architecture

This document is for people who change convy's code. It describes how the code is laid out and the rules that hold everywhere in it. How to use convy is in the README and docs/; why the main choices were made is in docs/decisions.

Bird's eye view

convy plays scenarios against an agent. For each attempt it opens a conversation with the agent, lets a simulated user talk to it, and asks a judge whether each of the scenario's claims holds. The simulated user and the judge are models; the agent is anything that implements a small interface. A run has a folder of its own: its specification and status in run.json, each attempt in attempts.jsonl as soon as it ends. The report is built from the runs.

convy run ──▶ RunJournal.play ──▶ Bench ──▶ Scenario.outcome ──▶ SimulatedUser ⇄ Conversation (agent)
                   │                                         └─▶ Judge
                   └──▶ results/<agent>/<run>/ ──▶ Runs ──▶ Report ──▶ results/<agent>/<run>/report.html
                                                        ├─▶ Comparison ──▶ results/compare/*.html
                                                        ├─▶ Index ──▶ results/index.html
                                                        └─▶ Tables ──▶ results/export/

Code map

All code is in src/convy/.

module	what is there
agent.py	the agent interface — Agent, Conversation, Message, Answer, Usage, NoUsage, an answer's steps Thought and Call, Reported, NotReported, AgentFailure — and its wrappers TimeLimited and QueuedAgent
busy.py	Busy, which an agent, model or judge raises when its service refused a request unprocessed; Patience, how long a participant waits for a busy service; Places, a participant's requests at once and per minute, fewer at once while its service is busy, which sends and retries them as its patience allows
http.py	JsonEndpoint (JSON over HTTP, raising Busy on 429 and 503) and JsonConnection, its open connection that a conversation keeps for all its turns; Tls, HttpFailure, and JsonAgent, an agent configured with a URL and a body template, and TokenPaths, where it finds an answer's tokens
model.py	the model interface Model, OpenAiModel, ModelFailure, Models, which holds the simulated user's model and the judge, and Contained, which turns any error of the simulated user's model into ModelFailure; PatientModel
env.py	Env, the base class for settings read from .env and the environment
dialog.py	Transcript, Turn, SimulatedUser, Finished; the judge interface Judge, ChatJudge, PatientJudge, and Checked, which turns any error or wrong answer of a judge into ModelFailure and gives a judge only what was said; a scenario's Graded claims; its decisions Verdict, Claim with Confidence or NoConfidence and Grade or NoGrade, and the attempts without them, Failed and NoVerdict
scenario.py	Scenario (read from YAML) and the attempt it plays, Outcome; Scenarios, Matching
bench.py	Bench, which runs scenarios against an agent, each participant within its Limits and with its Patience; Journal; a run's RunSpec, its status (Running, Finished, Interrupted) and RunJournal, which writes its folder and holds it while it plays, or raises RunInUse when another process does
report.py, pages/	Runs, which reads runs back; Summary, a run summed up into view records, each time and token number a Spread of its values, and Tally, the result of held out of judged; the pages: Report (a run), Comparison (two runs), Index (every run), each rendered through a Template, which inlines the shared parts in pages/ — base.*, icon.svg and the scenario window's window.* — into the page
tables.py	a consumer of runs: Tables, a run's seven tables, and Table, one of them with its rows as records, CSV and JSON Lines; the row types
fakes.py	FakeAgent, Echo, FakeModel, FakeJudge, MemoryJournal
project.py	Project: a user's project — it loads models.py and agents/*.py, with the project folder on sys.path so they import each other, finds scenarios and results, fingerprints the files a run starts with, writes the reports, the index, comparisons and exports, and copies the template; ProjectAgent; Settings, the limits in [tool.convy] of pyproject.toml, read as Pyproject, and Variables, the CONVY_* variables over it
cli.py	the convy command: InitCommand, RunCommand, ResumeCommand, ReportCommand, CompareCommand, ExportCommand; Printed, a journal that prints progress; Recorded, a run played as the command shows it, Ctrl+C included; Invalid, settings errors shown without the values read; Rebuilt, the report written again and shown with the runs it skipped
template/	the project that convy init copies

convy run runs models.py once per agent, so each agent gets fresh models; code at the top level of models.py runs that many times.

A user's project holds the files convy loads: models.py, agents/*.py, scenarios/*.yaml, .env, and results/, which convy writes. Other modules in it, such as the template's gateway.py, are what those files import.

The public API is convy.__all__ and the module convy.fakes; with them a run can be made from Python as well as from the command. Everything else may change without notice.

Invariants

convy follows the ideas of Elegant Objects by Yegor Bugayenko, in idiomatic Python. These rules hold in all of src/convy/.

1. A class stands for a thing in the problem, not an action: an agent, a conversation, a scenario, a judge, a journal. There are no *Manager, *Runner or *Parser classes, no static methods and no module-level functions with logic. The one function is main in cli.py, the command's entry point.
2. Objects do not change after construction. Classes are msgspec.Struct with frozen=True. A constructor stores its arguments and may reject a value the object cannot represent (ValueError); it never does work — no I/O, no computing, no calls to other objects. The check is in __post_init__, which msgspec also calls on decoding, so a scenario file gets it too; a check that depends on the situation, such as whether a certificate file exists, stays in the method that needs it.
3. An interface (typing.Protocol) exists only where there are two implementations or more. There are five: Agent, Conversation, Model, Journal and Judge.
4. New behaviour is a wrapper, not a flag or a subclass. TimeLimited(agent, 600) is the same agent with a time limit on opening a conversation, on each turn, and on closing. Only exceptions inherit, from Exception.
5. "No value" is an empty object, not None: NoUsage, NotReported, NoReason, NoVerdict. It is stored and read like the real value, with no type checks: msgspec tags it ({"type": "no_usage"}) and restores the right class. None is allowed only for an optional constructor argument that was not given, and for null in a type that mirrors outside JSON (rule 7).
6. Domain objects become JSON in one place, the run's folder. Objects have no json() or to_dict() methods. RunJournal writes with msgspec.json.encode; Runs reads back with msgspec.json.decode. The report's pages embed data of their own: Summary builds the view records for the browser from the runs; Template encodes them. Rows of Tables are not domain objects either but data for outside: Table gives them as records, CSV and JSON Lines, and Project.export writes them. Domain objects still have no such methods.
7. Outside data becomes the domain's types where it enters; only those types travel inside. Each entry turns what comes in into convy's own types at once. Scenarios: yamlrocks, then msgspec.convert into Scenario. The run's folder: msgspec.json.decode into RunFile and Outcome, an older format converted right there. An agent's answer: encoded and decoded with msgspec.json as the run's folder will, so a wrong one fails its turn instead of the run. A model's reply: read into the shape of an OpenAI chat completion. The judge model's reply: each decision read into the type its claim needs. A judge's own answer: Checked encodes and decodes it as the run's folder will, and wants one Claim per claim, a graded claim with one of its grades and the pass the scenario gives that grade; a judge gets only what was said. [tool.convy] of pyproject.toml: msgspec.toml.decode into Pyproject. The environment and the command line: pydantic-settings; the CONVY_* variables are then checked by msgspec as Settings. Past the entry nothing checks a type by hand: no raw dicts and no isinstance on what came in. A type that mirrors outside JSON may hold None for its null, and is turned into domain types at once. JSON of no fixed shape, a JsonAgent's body template and paths, which the user sets, is walked with match, and what is taken from it is converted on the spot. Pydantic is used only for settings; every other class is msgspec.
8. Every interface has a fake in fakes.py, part of the public API. A fake is a simple working object, not a mock. Fakes get no methods that exist only for tests; a fake may keep what it collected in public fields (outcomes, peak).
9. The library does not print or log. It returns what the command needs to show, such as Runs.broken(). Only cli.py prints and picks the exit code.
10. Types are checked on data, never on behaviour. match reads data: the closed unions — Message | Finished, Usage | NoUsage, Thought | Call, Reported | NotReported, int | NotReported (a count of Usage), Reason | NoReason, Verdict | Failed | NoVerdict, a claim's str | Graded, a run's status — and records such as Outcome. Data from outside is read into such types where it enters (rule 7), so this holds for it too. Objects with behaviour are called through their interface, not checked for type. There are two exceptions, both where convy meets the outside: Project checks the type of what a project file defines (isinstance(models, Models)), as outside data is checked where it enters; and main checks that stdout and stderr are text streams before it sets them to UTF-8.

Where convy departs from Elegant Objects:

• Data has public fields. Message, Usage, Answer, Turn, Verdict, Outcome and RunSpec are immutable records with public fields; accessor methods would be noise in Python.
• A conversation is opened with async with, Python's usual "open, use, close".
• Four kinds of objects change inside: a conversation (its session and history), fakes (they count), MemoryJournal, and Places (a participant's requests under way and the places it allows; a bench makes new ones for every run). They are plain classes, not frozen Structs, and their state never leaves them.
• The report's page data uses None (null in JSON) for numbers nobody reported: it is data for the browser, not part of the domain; and so do the tables' rows: a value the agent or the judge did not give, or one that does not apply to the row, is None, a missing value for pandas, polars and CSV.

The report

report.py makes every number and decision on the pages: the tiles, a scenario's result, a claim's count, the change between two runs and its colour, whether a change of pass rate is real, the groups and order of the rows. The scripts in pages/ only draw them — how a value looks, the filter, the scenario window, the theme, the address — and add data as text, never as HTML; agent text goes through a small Markdown subset built as DOM nodes. A metric is therefore tested with pytest, and the scripts stay thin enough to need no tests of their own. Each page is assembled from shared parts in pages/ — base.css, base.js and the tab's icon.svg for every page, window.css and window.js for the scenario window — and written as one standalone file.

The view records are internal and have no version: one convy version writes the template and the data into one file, and a page written earlier keeps working on its own. The durable formats are the run's folder and the tables' columns: a column may be added, and renaming or removing one is a breaking change; see decision 5 and decision 11.

Cross-cutting concerns

Concurrency. Conversations run at the same time, so the whole path is async: asyncio.timeout for time limits, asyncio.TaskGroup for concurrent work, and Places per participant for how much it does at once (decision 7), fewer while its service is busy (decision 8), and how often it sends (decision 9). Bench.run makes them in the event loop that waits for them and hands them to QueuedAgent, PatientModel and PatientJudge: the agent's place is held while a conversation is open, the user's and the judge's for one try of a request, so none is held while waiting for a busy service, and the judge decides after the agent's is let go. Nothing blocks the event loop except the run's append of one short line per attempt and the rewrite of its small run.json. Durations are measured with time.monotonic; Places keeps its moments in the event loop's time, which its sleeps follow. One process plays a run's folder at a time: RunJournal.play holds run.lock, which the system frees when the process ends, however it ends; taking it does not wait. Ctrl+C makes asyncio.run cancel the bench; RunJournal.play marks the run interrupted on that, or on any other error, and re-raises it.

Failure. Four kinds, one rule each:

• AgentFailure — the agent failed: the dialogue ends and the attempt is Failed without asking the judge, so a broken agent cannot pass on what it said before.
• Busy — a service refused a request before acting on it. The participant's Patience waits and sends it again; when it gives up, an agent's attempt has NoVerdict and stops as agent_busy, and a model's or a judge's Busy is a ModelFailure (decision 8).
• ModelFailure — the simulated user's or the judge's model failed, stayed busy, or gave an answer convy cannot use: an empty message, or a judge's answer without one decision per claim whose "pass" is true or false, or whose "grade" is one of the claim's; or raised any other error, which Contained (the user's model) turns into ModelFailure; or a judge that raised any error, or gave an answer that is not one Claim per claim with the grades the scenario allows, which Checked turns into ModelFailure. The attempt has NoVerdict and is left out of the pass rate.
• Configuration errors — raised while loading the project, or a run that cannot be resumed or that another process is playing (RunInUse), before any request: the command prints them and exits with code 2.

except Exception appears in four places: in scenario.py around the agent's code — opening a conversation, a turn, closing it — because a bug there must not stop a run; in Contained (model.py) around the simulated user's model's reply, so a bug in it is a ModelFailure, not the agent's failure or a crashed run; in Checked (dialog.py) around a judge's decide, for the same reason; and in cli.py around loading the project, where any error is a configuration error.

Retries live only in Places, one for each participant of a run. Only a request whose service refused it unprocessed is repeated, which its caller reports by raising Busy, so an agent never gets the same message twice: JsonConnection raises it for 429 and 503, while a 500, 502, 504 or a read timeout may come after the service acted on the request, and a connection that could not be made means no service at all, so each is a failure.

Secrets. Keys are SecretStr and never printed. A settings error is shown by field and message only, because pydantic's own text includes every value it read; Env also sets hide_input_in_errors, since an agent that reads its settings in conversation() fails with that error into the run's folder; an HTTP error shows the URL without its query, which may hold a key. Models are written to runs by name only, never as objects, because an object holds request headers with keys.

Trust. convy run imports the project's Python files and runs them in its own process; see the security policy. The other way round, nothing an agent says is trusted: the report shows dialogues only as text, never as HTML.

Dependencies. httpx2, msgspec, yamlrocks and pydantic-settings; why these is in decision 4. Adding a runtime dependency needs an ADR.