feat: layout blocks JSON

[mvanhorn]

Summary

Adds extract_layout_blocks / extract_layout_blocks_mem: run the existing Full-mode extract+convert pipeline and return a Firecrawl-compatible blocks payload for citation grounding. No layout model — the Markdown convert loop already classifies every fragment it emits (heading tier, list item, caption, code, table, image); it just never serialized those decisions. A new opt-in BlockRecorder (src/markdown/blocks.rs) captures them as they happen, so the hypothesis in the issue held and no re-derivation pass was needed.

Each block carries:

• markdownSpan: exact [start, end) byte offsets into the payload's own markdown string, recorded as the convert loop builds the document. Spans are ascending, non-overlapping, land on UTF-8 boundaries, and cover ~99% of the markdown on the fixture corpus (only inter-block separators fall outside).
• bbox: normalized 0–1 page space, top-left origin, from the existing TextItem / table / image coordinates and the page's CropBox/MediaBox.
• type: hosted names via heading tier 1 → title, tiers 2–6 → section_header (label H2..H6), body → text, list → list_item, caption → caption, code → code, table → table, image → picture. Furniture is stripped before conversion as today and is never invented as blocks.
• source: always native_text; confidence.layout / confidence.ocr stay null because no layout or OCR model produces a real score.

Default output is untouched. Recording is a parallel observation of the same conversion pass: with no recorder the pipeline is byte-identical (recording_does_not_change_default_markdown unit test), and all existing markdown snapshot tests pass unchanged.

Span exactness vs document-level postprocess. The document-wide cleanup pass (hyphenation, space collapsing, folio removal) rewrites bytes, which would invalidate recorded offsets. The payload therefore runs the same cleanup per recorded fragment and reassembles the markdown with the original inter-block separators, keeping spans exact. On 7 of 10 snapshot fixtures the payload markdown is byte-identical to process_pdf; the remaining three differ by 1–2 lines where document-level hyphenation joins a word across a block/page boundary (e.g. Green-Comp stays hyphenated) — inherent to guaranteeing exact spans, and documented on the API.

Surface

• Rust: extract_layout_blocks(path) / extract_layout_blocks_mem(bytes) returning LayoutBlocksResult { markdown, blocks, pdf_type, page_count }. Scanned/image-based PDFs (and garbage-text suppressions) return the classification with empty markdown and no blocks, mirroring process_pdf's trust decisions.
• CLI: new pdf2md --layout-blocks-json flag emitting {"schema_version":1, "pdf_type", "page_count", "markdown", "blocks":[{"type","label","page","bbox","markdownSpan","source","confidence":{"layout":null,"ocr":null}}]}, hand-rolled JSON like the existing --json/--items-json outputs. Default pdf2md output is unchanged.
• Python: extract_layout_blocks / extract_layout_blocks_bytes + LayoutBlock / LayoutBlocksResult classes, .pyi stubs, and docs/python.md, wired like extract_structure_elements.
• Node (napi): extractLayoutBlocks(buffer) with LayoutBlockJs / LayoutBlocksResultJs, README section, and test.mjs coverage. WASM intentionally omitted (root crate still checks for wasm32-unknown-unknown).

Testing

• cargo fmt / cargo fmt --all -- --check clean (root and napi crates).
• cargo clippy -- -D warnings and cargo clippy --features ocr -- -D warnings clean.
• cargo test and cargo test --features ocr: all pass (1038/1126 unit + 169 integration), including unchanged markdown snapshots.
• New tests: 9 recorder unit tests (src/markdown/blocks.rs), 4 integration tests including a synthetic lopdf document (title/paragraph/list geometry and top-left-origin ordering assertions), fixture grounding (span validity, ≥90% coverage, table blocks, H* labels), byte-equality with process_pdf_mem on the snapshot fixture, and the image-based-PDF empty payload.
• Python: maturin develop + pytest tests/test_python.py — 90 passed; the single failure (test_process_all_fixtures[encrypted-secret123.pdf]) is pre-existing on main (that parametrized test runs process_pdf without a password over every fixture).
• napi: napi build --platform + node test.mjs — all pass including the new extractLayoutBlocks span/bbox assertions.
• cargo check --target wasm32-unknown-unknown passes.
• Not run: the pdf-evals regression suite (bench.py test) — the sibling repo isn't available in this environment. Since the default pipeline is byte-identical without a recorder and all in-repo snapshots pass, no snapshot churn is expected.

Out of scope

ML layout models, default markdown changes, new language bindings (WASM), and the furniture audit report (#459).

---

Summary by cubic

Adds layout-block extraction across Rust, the CLI, Python, and napi. It runs the existing Full-mode conversion pipeline and returns markdown plus Firecrawl-compatible typed blocks for citation grounding, with exact byte spans and normalized page coordinates; default process_pdf output remains unchanged.

New Features

• Blocks cover headings, text, lists, captions, code, tables, and images with hosted types, page numbers, top-left 0–1 bboxes, native_text provenance, and null confidence.
• Adds pdf2md --layout-blocks-json, extract_layout_blocks / extract_layout_blocks_bytes, and synchronous extractLayoutBlocks(buffer) APIs; WASM remains unsupported.
• Layout-block payloads include image placeholders, so their markdown can differ from default output; fragment-level cleanup preserves exact spans, with only cross-boundary hyphenation differing on some fixtures.
• JS markdownSpan is f64 so offsets stay exact past 4 GiB; page-number-shaped list fragments are kept only when adjacent to another list item on the same page.
• Scanned and image-based PDFs follow existing trust decisions and return empty markdown and blocks.

^{Written for commit f758607. Summary will update on new commits.}

[cubic-dev-ai]

All reported issues were addressed across 15 files

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 10 files (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic</sub>

[mvanhorn]

Addressed the review findings in f758607.

markdownSpan was emitted as u32, so byte offsets into markdown past 4 GiB wrapped. It's f64 now — JS numbers stay exact well beyond that — with a regression test at the u32 boundary and a Number.isSafeInteger assertion in the napi smoke test.

On the list-item folio exemption: the blanket skip was too broad. Page-number removal decides by line isolation, and a single-item fragment is always isolated, so exempting every list item kept fragments the document-level pass would have dropped. The exemption now applies only when an adjacent list fragment on the same page makes the item non-isolated, which is the same conclusion the document-level pass reaches.

1040 lib tests and the napi test pass.

[cubic-dev-ai]

0 issues found across 3 files (changes from recent commits).

<sub>Shadow auto-approve: would require human review. Adds an opt-in layout-blocks extraction feature with a public API across Rust, Python, Node, and a CLI flag. Needs human review since it defines new API contracts and updates the core conversion loop; the pdf-evals suite hasn't run.

Re-trigger cubic</sub>