feat(datafabric): ask the user to disambiguate when similar entities match - #1084
milind-jain-uipath wants to merge 2 commits into
Conversation
…match The entity-query subgraph is non-interactive and the ENTITY SELECTION step told the model to always pick an entity, so with two similar entities in context it silently assumed one and ran SQL. Add a rule to the v1 planning prompt: when 2+ entities are equally plausible and the question doesn't disambiguate, return a clarifying question naming the candidates instead of calling execute_sql. The subgraph's router already terminates cleanly on a no-tool reply, so the clarifying question bubbles up to the outer agent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
🟡 Changes recommended
The shared v1 prompt affects ontology flows despite the stated scope, and the rendered instruction lacks a regression assertion.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds Data Fabric prompt guidance to request clarification when multiple entities are equally plausible, and bumps the package version.
Changes:
- Adds entity-disambiguation instructions to the v1 prompt.
- Updates version
0.18.3→0.18.4and synchronizes the lockfile.
File summaries
| File | Summary | Findings |
|---|---|---|
uv.lock |
Synchronizes the locked package version. | No final comments. |
src/uipath_langchain/agent/tools/datafabric_tool/prompts/v1.py |
Adds entity-disambiguation guidance. | Moderate (3 votes): the shared v1 template also affects the ontology path; separate the template or clarify the scope. Nit (3 votes): add a rendered-prompt regression assertion. |
pyproject.toml |
Updates the package version. | No final comments. |
Review details
- Files reviewed: 2/3 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| question requires fields from multiple entities. If two or more entities are \ | ||
| equally plausible matches for the data the question asks about (e.g. similarly \ | ||
| named entities, or entities with overlapping schemas) and the question does \ | ||
| not make clear which one to use, do NOT guess and do NOT silently pick one — \ | ||
| instead of calling ``execute_sql``, reply with a brief clarifying question \ |
There was a problem hiding this comment.
disambiguation is desirable in both the entity-set and ontology paths, so we're keeping the single shared template rather than forking it. Corrected the PR description, which previously (wrongly) said the ontology path was unchanged.
| question requires fields from multiple entities. If two or more entities are \ | ||
| equally plausible matches for the data the question asks about (e.g. similarly \ | ||
| named entities, or entities with overlapping schemas) and the question does \ | ||
| not make clear which one to use, do NOT guess and do NOT silently pick one — \ | ||
| instead of calling ``execute_sql``, reply with a brief clarifying question \ |
There was a problem hiding this comment.
Added in 7a0ae6c — test_build_includes_entity_disambiguation_rule asserts the rendered v1 prompt contains the rule and its no-tool directive (do NOT guess…, reply with a brief clarifying question, instead of calling \`execute_sql```), so a future template/rendering edit can't silently drop or join it.
Regression per PR review: assert the rendered v1 prompt contains the
disambiguation rule + its no-tool ("instead of calling execute_sql") directive,
so a future template/rendering edit can't silently drop or join it. Covers both
entity-set and ontology paths (both render through the default v1 strategy).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
What
When the Data Fabric context contains similar entities (similar names / overlapping schemas) and a query doesn't make clear which one to use, the agent currently silently assumes one and runs SQL. This adds a prompt rule so it instead asks the user which entity to use.
Why
The entity-query inner subgraph is non-interactive (bound only to
execute_sql, no HITL node), and the v1ENTITY SELECTIONstep told the model to always pick an entity ("Which entity answers this? … Prefer the fewest"). So with two plausible entities it guesses. Reported from a cross-source-join scenario where the agent assumed one of several similar entities.How (Option A — prompt-only, fits the existing architecture)
Extend
prompts/v1.pystep 1:The subgraph
routeralready returnsENDon an AIMessage with no tool call, andDataFabricTextQueryHandler.__call__returns that terminal AIMessage's text — so the clarifying question bubbles up to the outer conversational agent and is relayed to the user. No code/graph change needed.v1is the default prompt version (DEFAULT_PROMPT_VERSION), so this takes effect immediately.Scope
The rule lives in the shared
v1strategy template, so it applies to both the entity-set path and the ontology path (ontology_prompt_builder.build→build_sql_context(prompt_version=None)→ defaultv1). This is intentional — disambiguation is equally desirable for ontology-grounded queries, so we keep the single shared template rather than forking it. (Corrects an earlier draft that wrongly said the ontology path was unchanged.)Limits
Soft clarification: it's model judgment surfaced as a normal assistant message, not a structured HITL interrupt. A true interrupt-based flow would be a larger change since the subgraph is deliberately non-interactive.
Tests
ruffclean;pytest tests/agent/tools/test_datafabric_prompt_builder.py test_datafabric_subgraph.pypasses. Addedtest_build_includes_entity_disambiguation_rule(regression) asserting the rendered v1 prompt contains the rule + its no-tool directive, so a future template edit can't silently drop or join it. Version bumped0.18.3 → 0.18.4(+ lock) so it publishes.🤖 Generated with Claude Code