docs(mcp): recommend quoted Lucene values for exact matches and bare terms for body text (#3548)

Refs #3538

## What

Only descriptions and the query guide change; Lucene processing is untouched.

- **Query tool `where` / `whereLanguage` descriptions** (`packages/api/src/mcp/tools/query/schemas.ts`):
  - for an exact match, quote the value: `SpanKind:"Server"` compiles to `=`, so a sort-key column such as `ServiceName` can use the primary key (SQL `=` is still listed);
  - for text in the log body, write the term or phrase without a field name. That uses the source's token/text index; `Body:term`, `Body LIKE` and `Body ILIKE` read the whole column, and `Body:"…"` is exact equality on the whole body.
- **Query guide prompt** (`packages/api/src/mcp/prompts/dashboards/content.ts`): the same two points in the SUBSTRING MATCHING section and in common mistake 15; the "Contains" example now uses `StatusMessage` instead of `Body LIKE '%…%'`.
- `prompts.test.ts` asserts both points are in the guide.

## Why

Agents read "for exact match, use SQL" and mostly kept writing `ServiceName:x`, which matches `x-jobs` too and skips the primary key (one hour of logs: 108.9M vs 1.65M rows read). For body text, the descriptions pointed to `LIKE`, which can't use the index (27.7 GiB vs 121 MiB for the same matches). Details in #3538.

What each recommended form compiles to on `main`:

| Lucene | SQL |
|---|---|
| `ServiceName:api` | `ServiceName ILIKE '%api%'` |
| `ServiceName:"api"` | `ServiceName = 'api'` |
| `timeout` | `hasAllTokens(lower(Body), lower('timeout'))` (with a text index) |
| `"connection refused"` | `hasAllTokens(lower(Body), lower('connection refused')) AND lower(Body) LIKE lower('%connection refused%')` |
| `Body:"connection refused"` | `Body = 'connection refused'` |

The optional hint ("`ServiceName:api` matched 2 values") is not part of this PR; happy to follow up if you want it.

## Tests

- `npx jest src/mcp --testPathIgnorePatterns '\.int\.test\.ts$'` in `packages/api`: 25 suites, 484 tests pass.
- `tsc --noEmit`, `eslint` (no new warnings) and `prettier` clean.

Author: bcanvural

.changeset/mcp-lucene-exact-match.md

@@ -0,0 +1,7 @@
+---
+'@hyperdx/api': patch
+---
+
+docs: MCP tool descriptions and the query guide now recommend the quoted Lucene
+form `field:"value"` for exact matches, and bare terms or phrases for log body
+text, so agents keep the primary key and the text index

packages/api/src/mcp/__tests__/dashboards/prompts.test.ts

@@ -175,8 +175,13 @@ describe('MCP Dashboard Prompts', () => {
       const luceneBody = prompt.slice(luceneIdx);
       expect(luceneBody).toMatch(/SUBSTRING MATCHING/);
       expect(luceneBody).toMatch(/ilike\(field, '%value%'\)/);
-      // The canonical fix is SQL with =.
+      // The canonical fix is SQL with =, or a quoted Lucene value.
       expect(luceneBody).toMatch(/SpanKind = 'Server'/);
+      expect(luceneBody).toMatch(
+        /SpanKind:"Server" compiles to SpanKind = 'Server'/,
+      );
+      // Body text goes through the token/text index as a bare term, not Body:term.
+      expect(luceneBody).toMatch(/bare Lucene term or phrase/);
       // Wildcard limitation is documented alongside substring matching.
       expect(luceneBody).toMatch(/prefix-within-substring/);
       expect(luceneBody).toMatch(/LIKE 'api%'/);

packages/api/src/mcp/__tests__/query.test.ts

@@ -28,6 +28,8 @@ import {
   getMetricSelectIssues,
   mcpQuantileLevelSchema,
   validateMetricSelectItems,
+  whereLanguageSchema,
+  whereSchema,
 } from '@/mcp/tools/query/schemas';
 import { resolveOrderBy } from '@/mcp/tools/query/table';
 import { getErrorCategory } from '@/mcp/utils/errors';
@@ -1414,3 +1416,17 @@ describe('mcpQuantileLevelSchema', () => {
     expect(optional.safeParse('0.97').success).toBe(false);
   });
 });
+
+describe('where / whereLanguage descriptions', () => {
+  it('steer exact matches to the quoted Lucene form and body text to bare terms', () => {
+    expect(whereSchema.description).toContain(
+      'quote the value: SpanKind:"Server"',
+    );
+    expect(whereSchema.description).toContain(
+      'BODY TEXT: write the term or phrase without a field name',
+    );
+    expect(whereLanguageSchema.description).toContain(
+      'quote the value (field:"value", case-sensitive)',
+    );
+  });
+});

packages/api/src/mcp/prompts/dashboards/content.ts

@@ -862,7 +862,7 @@ Used in the "where" field of select items and search tiles.
   OR:                 level:error OR level:fatal
   NOT:                NOT level:debug
   Wildcard:           ServiceName:front*
-  Phrase:             Body:"connection refused"
+  Phrase:             "connection refused"   (default search column, e.g. the log body)
   Exists:             _exists_:http.route
   Range (numeric):    Duration:>1000000000
   Range (inclusive):  http.status_code:[400 TO 499]
@@ -883,13 +883,22 @@ Lucene field:value does NOT mean exact equality. It translates to ilike(field, '
      ServiceName:api*    does NOT mean "starts with api". It means "contains a substring starting with api".
                          In practice this often works for prefix matching, but it is NOT equivalent to SQL LIKE 'api%'.
 
-  For exact matching or true prefix/suffix/contains patterns, use SQL instead:
+  For exact matching in Lucene, quote the value: SpanKind:"Server" compiles to SpanKind = 'Server'
+  (case-sensitive, unlike the unquoted form).
+
+  For true prefix/suffix/contains patterns, use SQL instead:
      Exact:    whereLanguage: "sql", where: "SpanKind = 'Server'"
      Prefix:   whereLanguage: "sql", where: "ServiceName LIKE 'api%'"
-     Contains: whereLanguage: "sql", where: "Body LIKE '%timeout%'"
+     Contains: whereLanguage: "sql", where: "StatusMessage LIKE '%timeout%'"
      Regex:    whereLanguage: "sql", where: "match(ServiceName, '^api-v[0-9]+')"
 
-  When Lucene substring matching is FINE: free-text columns like Body, StatusMessage where substring is the intent.
+  For text in the log body, write a bare Lucene term or phrase (timeout, "connection refused"). It searches the
+  source's default search column (the body for logs) and uses its text index when the source has one. With an
+  index it matches whole tokens (timeout does not match ConnectionTimeoutException); for a substring inside a word, use
+  SQL ILIKE. Body:timeout, Body LIKE and Body ILIKE read the whole column, and Body:"..." is exact equality on
+  the whole body.
+
+  When Lucene substring matching is FINE: free-text columns like StatusMessage where substring is the intent.
   When Lucene substring matching is WRONG: enum-like columns (SpanKind, SeverityText, StatusCode, ServiceName) where you want exact or true prefix match.
 
 *** MAP-ATTRIBUTE GOTCHA ***
@@ -923,7 +932,7 @@ Alternative to Lucene. Set whereLanguage: "sql" when using SQL syntax.
   Basic:              SeverityText = 'error'
   AND/OR:             ServiceName = 'api' AND StatusCode = 'STATUS_CODE_ERROR'
   IN:                 ServiceName IN ('api', 'web', 'worker')
-  LIKE:               Body LIKE '%timeout%'
+  LIKE:               StatusMessage LIKE '%timeout%'
   Comparison:         Duration > 1000000000
   Map access:         SpanAttributes['http.status_code'] = '500'
 
@@ -1562,8 +1571,9 @@ Example: find top patterns for production services over the last 4 hours:
     Wrong:   where: "SpanKind:Server"             (whereLanguage: "lucene"; translates to ilike(SpanKind, '%Server%') and matches "ServerStreaming", "InternalServer", etc.)
     Wrong:   where: "ServiceName:api*"            (whereLanguage: "lucene"; NOT a true prefix match -- it is prefix-within-substring)
     Correct: where: "SpanKind = 'Server'"          (whereLanguage: "sql"; exact match)
+    Correct: where: 'SpanKind:"Server"'            (whereLanguage: "lucene"; a quoted value is an exact match)
     Correct: where: "ServiceName LIKE 'api%'"      (whereLanguage: "sql"; true prefix match)
-    Lucene field:value is a substring match (ilike with surrounding wildcards), not exact equality. Lucene field:value* is prefix-within-substring, not SQL LIKE 'value%'. Fine for free-text columns (Body, StatusMessage); wrong for enum-like columns where exact match matters (SpanKind, SeverityText, StatusCode) and wrong when true prefix matching is needed. Use SQL for exact match (=), prefix (LIKE 'x%'), contains (LIKE '%x%'), or regex (match(col, 'pattern')).
+    Lucene field:value is a substring match (ilike with surrounding wildcards), not exact equality. Lucene field:value* is prefix-within-substring, not SQL LIKE 'value%'. Fine for free-text columns (StatusMessage; for Body use a bare term or phrase, which uses the index); wrong for enum-like columns where exact match matters (SpanKind, SeverityText, StatusCode) and wrong when true prefix matching is needed. Use SQL for exact match (=), prefix (LIKE 'x%'), contains (LIKE '%x%'), or regex (match(col, 'pattern')).
 
 16. Forgetting that map-attribute values are often empty strings
     Wrong:   groupBy on SpanAttributes['db.collection.name'] without filtering empty values; the result has an empty-string bucket alongside the real ones, which renders as a blank row in tables and an unlabelled slice in pies.

packages/api/src/mcp/tools/query/schemas.ts

@@ -20,7 +20,14 @@ const WHERE_DESCRIPTION =
   'WRONG: level = "error"               (SQL syntax with whereLanguage:"lucene")\n\n' +
   'SUBSTRING TRAP: Lucene field:value matches ANY row containing "value" as a substring, not exact equality.\n' +
   '  SpanKind:Server matches "Server", "ServerStreaming", "InternalServer", etc.\n' +
-  "  For exact match, use SQL: SpanKind = 'Server'";
+  '  For exact match, quote the value: SpanKind:"Server" (compiles to a case-sensitive =, so a sort-key\n' +
+  "  column such as ServiceName can use the primary key), or use SQL: SpanKind = 'Server'\n\n" +
+  'BODY TEXT: write the term or phrase without a field name, e.g. timeout or "connection refused".\n' +
+  "  That searches the source's default search column (the body for logs) and uses its text index\n" +
+  '  when the source has one. With an index it matches whole tokens: timeout does not match ConnectionTimeoutException.\n' +
+  "  For a substring inside a word, use SQL ILIKE '%timeout%' (reads the whole column).\n" +
+  '  Body:timeout, Body LIKE and Body ILIKE read the whole column, and Body:"..." is exact equality on the\n' +
+  '  whole body.';
 
 const WHERE_LANGUAGE_DESCRIPTION =
   'Language of the "where" filter. The syntax in "where" MUST match this setting.\n' +
@@ -30,7 +37,8 @@ const WHERE_LANGUAGE_DESCRIPTION =
   'and map attributes via dot notation. Use "sql" for IN(...) lists, complex expressions, or function calls.\n' +
   'IMPORTANT: Lucene field:value is a SUBSTRING match (ilike), not exact equality. ' +
   'field:val* is prefix-within-substring, not a true prefix match. ' +
-  "For exact matching or reliable wildcards, use SQL: WHERE field = 'value' or WHERE field LIKE 'val%'.\n\n" +
+  'For exact matching, quote the value (field:"value", case-sensitive) or use SQL: WHERE field = \'value\'. ' +
+  "For reliable wildcards, use SQL: WHERE field LIKE 'val%'.\n\n" +
   'Common mistake: writing Column:value (Lucene) but setting whereLanguage to "sql". ' +
   'If your filter uses colon syntax, use "lucene".';