Summary
evidence validate reports valid: true and exits 0 when a page contains a critical Markdoc parse error — e.g. an undefined tag. The same page renders a page-level error box in serve, so the command gives a false all-clear for pages the app renders as broken.
Environment
- Evidence CLI v0.10.1
- No warehouse connection needed (syntax-only mode)
Steps to reproduce
Actual
{"valid":true,"fileCount":1,"errorCount":0,"warningCount":0}
exit code 0. The per-file output still contains the error — it is just not counted:
{"line":1,"endLine":1,"severity":"critical","message":"Undefined tag: 'unknown_tag'","id":"tag-undefined"}
Expected
Nonzero exit for critical parse errors. The CLI's human report already prints critical entries in red as errors, and serve renders them as a page-level error box.
Root cause
cli/src/routes/api/validate/+server.ts maps every validationErrors entry to the output (severity: err.error.level), but the aggregation only counts 'error':
totalErrors += errors.filter((e) => e.severity === 'error').length;
totalWarnings += errors.filter((e) => e.severity === 'warning').length;
Markdoc classifies structural errors (tag-undefined, missing-opening, tag-placement-invalid, ...) as level: 'critical', which falls into neither bucket:
errorCount: 0, warningCount: 0, valid: true.
Suggested fix
totalErrors += errors.filter((e) => e.severity === 'error' || e.severity === 'critical').length;
Impact
Medium. Anyone relying on evidence validate's exit code gets a false green on pages that serve renders with an error box. A "treat warnings as errors" style option would not help — critical entries are counted as neither errors nor warnings.
Summary
evidence validatereportsvalid: trueand exits0when a page contains a critical Markdoc parse error — e.g. an undefined tag. The same page renders a page-level error box inserve, so the command gives a false all-clear for pages the app renders as broken.Environment
Steps to reproduce
{% unknown_tag /%}Actual
{"valid":true,"fileCount":1,"errorCount":0,"warningCount":0}exit code
0. The per-file output still contains the error — it is just not counted:{"line":1,"endLine":1,"severity":"critical","message":"Undefined tag: 'unknown_tag'","id":"tag-undefined"}Expected
Nonzero exit for critical parse errors. The CLI's human report already prints
criticalentries in red as errors, andserverenders them as a page-level error box.Root cause
cli/src/routes/api/validate/+server.tsmaps everyvalidationErrorsentry to the output (severity: err.error.level), but the aggregation only counts'error':Markdoc classifies structural errors (
tag-undefined,missing-opening,tag-placement-invalid, ...) aslevel: 'critical', which falls into neither bucket:errorCount: 0,warningCount: 0,valid: true.Suggested fix
Impact
Medium. Anyone relying on
evidence validate's exit code gets a false green on pages thatserverenders with an error box. A "treat warnings as errors" style option would not help —criticalentries are counted as neither errors nor warnings.