Skip to content

evidence validate exits 0 on critical markdown parse errors #3340

Description

@hpcnt-jerry-l

Summary

evidence validate reports valid: true and exits 0 when a page contains a critical Markdoc parse error — e.g. an undefined tag. The same page renders a page-level error box in serve, so the command gives a false all-clear for pages the app renders as broken.

Environment

  • Evidence CLI v0.10.1
  • No warehouse connection needed (syntax-only mode)

Steps to reproduce

{% unknown_tag /%}
evidence validate --json

Actual

{"valid":true,"fileCount":1,"errorCount":0,"warningCount":0}

exit code 0. The per-file output still contains the error — it is just not counted:

{"line":1,"endLine":1,"severity":"critical","message":"Undefined tag: 'unknown_tag'","id":"tag-undefined"}

Expected

Nonzero exit for critical parse errors. The CLI's human report already prints critical entries in red as errors, and serve renders them as a page-level error box.

Root cause

cli/src/routes/api/validate/+server.ts maps every validationErrors entry to the output (severity: err.error.level), but the aggregation only counts 'error':

totalErrors += errors.filter((e) => e.severity === 'error').length;
totalWarnings += errors.filter((e) => e.severity === 'warning').length;

Markdoc classifies structural errors (tag-undefined, missing-opening, tag-placement-invalid, ...) as level: 'critical', which falls into neither bucket:
errorCount: 0, warningCount: 0, valid: true.

Suggested fix

totalErrors += errors.filter((e) => e.severity === 'error' || e.severity === 'critical').length;

Impact

Medium. Anyone relying on evidence validate's exit code gets a false green on pages that serve renders with an error box. A "treat warnings as errors" style option would not help — critical entries are counted as neither errors nor warnings.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions