test(conformance): add agent SDK parser suites and run them in the Python SDK - #2713
Merged
Merged
Conversation
Covers the agent SDK described in blueprints/modules/a2ui_agent.blueprint.md: catalog providers, catalog resolution, catalog transformers, the request processor, skills, and the two inference formats. Every case is written against v1.0. agent/ holds the suites no format decides for itself. agent/direct_json/ and agent/express/ hold the rest, written in each format's own notation, so an implementer adding a format can run one folder and know when they are done. Streaming is the exception: the Express parser does not implement parse_chunk, so only direct_json has a streaming suite. The suites for the earlier agent interface move to agent/legacy/, with the Python and Kotlin harness paths updated. They stay until those SDKs move to the blueprint interface. Catalog fixtures move to test_data/catalogs/, and prompt examples are fixture files under test_data/examples/ referenced by path. conformance_schema.json gains the vocabulary these suites need, and conformance/README.md documents how to write cases in it. An agent case states no protocol version of its own: the version comes from the catalog document, or from what a provider is constructed with. The blueprint leaves several questions open, and the suites answer them rather than leave them to each implementation. Each answer is argued in the header of the file that relies on it, so a reader can see what was decided and why.
The Express proposal showed a standalone call compiling to a top level functionCallId beside a callFunction. agent_to_renderer.json has no such message: CallRendererFunctionMessage nests both inside callRendererFunction, and marks catalogId required on the call. A payload in the shape the proposal showed fails validation. Both compilers implement the proposal faithfully, so both emit an invalid message. The normative specification/v1_0/docs/a2ui_protocol.md had the nesting right all along, which is why this went unnoticed. Fixing the proposal first so the compilers have something correct to follow.
The vocabulary said unwrap_response, wrap_response, compile_payload and decompile_payload. The blueprint's Parser declares unwrap, wrap, compile and decompile. parse_response and parse_chunk already matched, so four of the six names differed from the method each one tests, for no reason beyond how they were first written. Renames the actions and the test definitions that go with them, so a reader moving between a case and the interface it exercises does not have to translate. The three remaining agent actions cannot follow: the action enum is flat across core/, agent/ and extensions/, and load, generate_prompt and transform are either taken or say nothing on their own. They keep their qualified names. test_unwrap_response_without_tags_is_one_text_part keeps its name. That is English rather than the old action, and its Express counterpart is spelled the same way.
response_parser.yaml held five different calls in one file, and the Express one
had grown to 74 cases behind a 135 line header covering all of them. Splits each
format's suite into three, one per call:
response_parser.yaml where a payload begins and ends: unwrap, wrap,
parse_response
compiler.yaml what a payload means
decompiler.yaml how messages are written back
The blueprint declares all five on one Parser interface, so this is three files
for one interface. They are split by call because each has rules of its own, and
the Express compilation rules alone run to a page. Each header was rewritten
rather than copied, so a rule sits with the cases that depend on it and there is
no second copy to drift: compiler.yaml owns the error categories, the Express id
scheme, the grammar facts and the positional argument order, and decompiler.yaml
owns only the two rules that belong to its direction.
Also ports the behaviour the Python Express unit tests covered and the suites
did not. 30 cases: the notation the grammar admits that no case exercised
(comments, semicolons, trailing commas, a statement spanning lines, a bare $, a
string holding an equals sign), two surfaces in one block, several checks in one
list, seven compile errors, and seven decompile cases below the message level,
where coverage was thinnest.
Two behaviours the unit tests assert were deliberately not ported. ?and([...])
is a Python extension the Express specification does not describe, so
test_compile_express_several_checks_in_one_list covers the spec-backed list form
instead. An unregistered function compiling rather than failing contradicts
catalog validation, so the suite rules the other way.
conformance_schema.json gains expect_present for a value the format must supply
but the protocol does not dictate, such as the message on a check written
without one. Pinning the wording would freeze one implementation's English into
the protocol.
Case count is unchanged across the split: 322 agent cases before, 322 after.
The runner loaded only the legacy suites, so agent/express/ and
agent/direct_json/ were inert YAML. Wires compiler.yaml and decompiler.yaml for
both formats: 67 cases, 45 passing, 21 xfail, 1 skipped. Direct JSON passes all
ten of its cases.
The adapter is thin, because this SDK already implements the blueprint Parser:
load the catalog a case names from conformance/test_data/, build the format's
parser, call compile or decompile. Two things the suites leave to a harness and
this one supplies: the v1.0 validation experiment, without which no v1.0 catalog
can build a validator, and the surface id, which the suites fix as
default_surface and ExpressParser defaults to main.
A case this SDK does not satisfy is marked xfail(strict=True) with the reason,
so fixing the implementation fails the marker rather than passing quietly. No
case was weakened to make it pass. All 21 are Express, and five defects account
for them:
- Three catalog violations compile clean. A component the catalog does not
declare is dropped, leaving an empty surface with no diagnostic.
- A2uiCompilationError descends from Exception rather than A2uiError, so no
error category can match it. Eight cases.
- Four decompiler defects: a standalone updateDataModel writes no surface
line, callRendererFunction decompiles to the empty string, an updateComponents
block cannot be recompiled, and two spellings its own compiler rejects.
- Inline component ids are a counter rather than parent plus property.
- An Event with no context compiles context: {} rather than omitting it.
One case is skipped rather than marked: a parser holds a single catalog, so a
block targeting a second one by id has no API to run through.
Adds the third pair, response_parser.yaml for both formats: 34 cases, 15
passing and 19 marked. The runner now carries 101 conformance cases, 60 passing,
40 xfail, 1 skipped.
One test function dispatches on the call a case names, since the file holds
three: unwrap into ordered raw parts, wrap parts back into a response, and
parse_response for both at once. The unwrap and wrap cases carry no catalog,
because neither call consults one, so they get the simplified fixture and never
read it.
Five defects account for the 19:
- A part carries text and payload together. The suites fix one or the other
per part, so a response of text, block, text, block, text unwraps to three
parts here where five are expected. Four cases, evenly split between the
formats, which makes it a disagreement about the protocol rather than a bug
in one parser.
- wrap is wrap_decompiled_blocks and takes raw strings rather than parts, so
it has nowhere to put a text part and always writes a tagged block. Seven
cases. The harness calls it with the raw blocks a case names and lets the
text go missing, rather than assembling the response itself, which would be
testing the harness.
- Direct JSON unwrap raises where the suites return parts: no tags, an empty
response, and an unterminated block. Four cases. The third matters most,
since raising throws away the partial block a streaming agent needs.
- parse_response catches the compiler's A2uiParseError and re-raises it as
A2uiCompilationError, so the category is lost at the entry point an agent
actually calls. The direct JSON compiler raises the right category and the
compiler suite passes on the same input.
- parse_response takes no wrapped argument, so a response known to carry no
sentinel tags cannot be handed to the compiler whole.
Seven tests whose every assertion is made by a conformance case that passes
today:
test_format_string_and_actions nested_function_call,
event_action_with_context
test_map_variable_inlining map_variable_is_inlined_where_it_is_used
test_skipped_and_omitted_arguments placeholder_skips_an_optional_argument,
trailing_optional_arguments_may_be_omitted
test_semicolons_and_trailing_commas_ semicolons_and_trailing_commas,
and_line_continuation statement_spans_several_lines
test_compilation_delete_surface_kwargs delete_surface_with_a_keyword_argument
test_compilation_multi_surface two_surfaces_in_one_block
test_decompile_delete_surface decompile_express_delete_surface
Nothing stopped being executed: each replacement is a case that passes rather
than one marked xfail.
That rule is what keeps the rest. A test whose conformance counterpart is xfail
is the only executing coverage of that path until the defect is fixed, so it
stays; the strict marker will say when it can go. So do the tests of what
conformance deliberately does not fix: error classes and their attributes, the
wording of a synthesised check message, thread safety, catalog polymorphism,
CatalogSchemaHelper internals, and v0.9 and v0.9.1 targeting, which the suites
do not cover at all.
Both module docstrings now say where the coverage moved, so it is not added back
here.
A compile failure left the SDK as A2uiCompilationError, which descended from Exception and so sat outside the A2uiError hierarchy every other failure in the SDK belongs to. A caller could not ask whether a block was rejected because it could not be read or because the catalog refused what it named, and the eight conformance cases that draw exactly that line had no class to match. A2uiCompilationError now descends from A2uiError, and gains two subclasses that also descend from A2uiParseError and A2uiValidationError. Choosing between them needs the compiler to keep the two apart, so the Express errors are grouped under ExpressParseError and ExpressValidationError, and the failures that were raised as bare ValueErrors join whichever group they belong to: a value outside an enum and a standalone call the target version has no message for are validation failures, while a block the grammar rejects and a malformed _template call are parse failures. The parser then picks the class by type test rather than by reading the message. Also fixes a NameError in ExpressUndefinedChildError, which referenced an undefined help_message and so could never be constructed.
parse_response caught everything the compiler raised and, unless it was already an A2uiCompilationError, re-raised it as one. Direct JSON raises A2uiParseError from its own reader, so a response whose payload could not be read reached the caller as an uncategorised compilation error and the conformance case asking for a parse error had nothing to match. An A2uiError now travels out as it came, carrying the parts read before it, and only an exception from outside the hierarchy is wrapped.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive set of conformance test suites for the A2UI agent SDK, covering catalog providers, resolution, transformers, and format-specific (Direct JSON and Express) compilers, decompilers, prompt generators, and response parsers. To support this, the Python SDK's Express compiler and parser are updated to raise more specific error types (ExpressParseError and ExpressValidationError), which map to new A2uiCompilationParseError and A2uiCompilationValidationError exceptions. Additionally, legacy tests are moved or refactored, and documentation is updated to reflect the new test structure and correct RPC message schemas. Feedback on the changes suggests using a 'version at least' check (e.g., >=) instead of strict membership checks when verifying protocol versions for feature availability to ensure forward compatibility.
nan-yu
marked this pull request as ready for review
September 21, 2026 20:22
- Port SemVer utilities and tests from v1_0 branch into a2ui_core - Add A2uiProtocolVersion / ProtocolVersion enum in a2ui_core.schema - Export semver utilities from a2ui.core.common - Use is_at_least_version with ProtocolVersion.V1_0 in ExpressCompiler
…ed output - Add delete_pointer helper to remove JSON Pointer paths from compiled payload - Strip expect_present pointers after presence assertion so assert compiled == test_case["expect"] is verified
- Remove unused conformanceDir and test_data directory scans from jsonDirs - Remove unused SIMPLIFIED_CATALOG_V09 constant and dead schemaMappings entries
- Accept details: list[A2uiErrorDetail] | None in A2uiCompilationError.__init__ - Forward details to super().__init__(message, details=details) - Pass error details when wrapping compilation exceptions in ExpressParser
…Error - Remove A2uiCompileError from a2ui_core in Python, Dart, Swift, and blueprints - Keep A2uiCompilationError in agent SDK and map to CompilationError category - Update category enum in conformance_schema.json to CompilationError - Remove outdated CompileError documentation in conformance/README.md and compiler headers - Update CHANGELOGs for python a2ui_core, python a2ui_agent, and dart a2ui_core
gspencergoog
approved these changes
Sep 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds conformance suites for the agent SDK blueprint's
Parserinterface, runsthem against the Python agent SDK, and fixes the defects the runner exposed that
could be fixed without changing what the suites say.
The suites are written against the interface in
blueprints/modules/a2ui_agent.blueprint.mdrather than against any one SDK, sothey describe what an implementation of that interface owes a caller. Where the
Python SDK disagrees with a suite, the case is recorded as a known gap with a
one-line reason rather than weakened or deleted.
Changes
Conformance suites
New suites under
conformance/agent/, one per format, named after theParsercall each exercises:
express/compiler.yaml,express/decompiler.yaml,express/response_parser.yaml,express/prompt_generator.yamldirect_json/compiler.yaml,direct_json/decompiler.yaml,direct_json/response_parser.yaml,direct_json/prompt_generator.yaml,direct_json/response_streaming.yamlrequest_processor.yaml, covering catalog resolution and message processingCase actions are named after the interface methods they call:
compile,decompile,wrap,unwrap,parse_response. Express has no streaming suitebecause the format does not stream.
The three existing suites are written against the SDKs' current interface rather
than the blueprint's, so they move to
conformance/agent/legacy/and stay thereuntil the SDKs move over. Nothing in them changed.
Catalog fixtures move to
conformance/test_data/catalogs/and examples toconformance/test_data/examples/. Cases name a fixture by path instead ofinlining it, so the same catalog can be shared and diffed. New fixtures cover a
forms catalog, a catalog with a property that refuses data binding, a catalog
with no id, and a malformed catalog.
conformance_schema.jsongains the new actions, theexpect_presentassertionfor cases that pin a field's presence rather than its value, and the argument
shapes the new suites use. All catalog fixtures
$refintocommon_types.jsonrather than redefining protocol common types.
Python agent SDK
tests/conformance/test_conformance.pyruns the compiler, decompiler andresponse parser suites: 101 cases, of which 69 pass, 31 are recorded as known
gaps, and 1 has no API to call at all. Each gap carries a one-line reason and is
an
xfail(strict=True), so fixing one fails the build until its entry isremoved.
Seven Express unit tests are removed. Each was deleted only because every
mechanism it exercised is now pinned by a conformance case that passes today; a
test whose conformance counterpart is a known gap was kept, because it is then
the only executing coverage of that path.
Fixes the runner exposed
A2uiCompilationErrordescended fromException, outside theA2uiErrorhierarchy the rest of the SDK raises, so a caller could not ask whether a block
was rejected because it could not be read or because the catalog refused what it
named. It now descends from
A2uiErrorand gains two subclasses that alsodescend from
A2uiParseErrorandA2uiValidationError. Telling the two apartneeds the compiler to keep them apart, so the Express errors are grouped under
ExpressParseErrorandExpressValidationError, and the failures that wereraised as bare
ValueErrors join whichever group they belong to.Parser.parse_responsecaught everything the compiler raised and re-raised itas
A2uiCompilationError, which discarded the category Direct JSON's reader hadalready set. An
A2uiErrornow travels out unchanged, carrying the parts readbefore it.
ExpressUndefinedChildErrorpassed an undefinedhelp_messageto its baseclass and would have raised
NameErroron construction. Nothing constructs it.specification/proposals/express/a2ui_express.mdshowed a standalone functioncall compiling to a top-level
functionCallIdandcallFunction, which noschema in
specification/v1_0/jsonadmits. Corrected to thecallRendererFunctionenvelope.Impact and risks
An Express compile failure now raises a
ValueErrorsubclass where itpreviously raised a bare
Exceptionsubclass. Existingexcept Exceptionandexcept A2uiCompilationErrorhandlers are unaffected; the change only widenswhat else will catch it.
The known gaps are strict xfails by intent. Fixing any one of them fails the
build until the entry is removed from
KNOWN_GAPS, which keeps the list honest.prompt_generator.yamlandrequest_processor.yamlare not yet wired into anyrunner. The Kotlin harness's three updated path constants in
ConformanceTest.ktfollow the fixture moves but were not run.
No renderer, Swift, or web behaviour changes.
Testing
agent_sdks/python/a2ui_agent:uv run pytest tests/gives 662 passed,1 skipped, 31 xfailed.
agent_sdks/python/a2ui_core:uv run pytestgives 210 passed.uv run pytest conformance/tests/gives 25 passed, whichvalidates every suite against
conformance_schema.json../scripts/fix_format.sh --checkpasses.point, so the branch bisects cleanly.