fix(markdown): recognize extractor bullet glyphs (▪ ‣ ◆) as list items by shahidbeig-a11y · Pull Request #476 · firecrawl/pdf-inspector

shahidbeig-a11y · 2026-08-31T18:20:26Z

Fixes #475

Summary by cubic

Recognizes additional bullet glyphs (▪ ‣ ⁃ and related) as list items in markdown extraction, so PDF lists using them are no longer emitted as plain paragraphs or misclassified as headings. Large squares and diamonds (■ □ ◆ ◇) stay out of list detection because documents use them as decorative heading prefixes, but remain layout-only gutter markers so list columns still reject spurious column splits. Fixes #475.

Shares BULLET_GLYPHS and is_standalone_bullet_glyph across list detection, formatting, standalone bullet merging, the layout marker guard, and chart-adjacent label filtering.
Keeps - and * as separate checks and excludes middle dot, en dash, and em dash.
Expands tests covering extended glyphs, negative cases, heading rejection, chart-padding guards, and a square-bullet merge stream-order regression.

^{Written for commit 0fb69bb. Summary will update on new commits.}

Align is_list_item(), starts_with_bullet_marker(), and format_list_item() with the bullet glyph set already used by extractor/layout.rs and extractor/underline.rs. PDF lists using ▪, ‣, ◆ (and related markers) were emitted as plain paragraphs and could be misclassified as headings. Closes firecrawl#475 Co-authored-by: shahidbeig-a11y <shahidbeig-a11y@users.noreply.github.com>

Add BULLET_GLYPHS shared constant and use it in list detection, formatting, and standalone bullet merging. Keeps - and * as separate checks; excludes middle dot and dash glyphs. Fixes firecrawl#475 Co-authored-by: shahidbeig-a11y <shahidbeig-a11y@users.noreply.github.com>

Add explicit ▪/‣/◆ detection and format tests, middle-dot negative format case, numbered list assertions, and heading rejection test. Co-authored-by: shahidbeig-a11y <shahidbeig-a11y@users.noreply.github.com>

cubic-dev-ai

1 issue found and verified against the latest diff

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/markdown/classify.rs">

<violation number="1" location="src/markdown/classify.rs:4">
P2: When a PDF places `‣` or `⁃` in standalone marker items, column detection can mistake the marker gutter for a real column because its marker guard does not know these newly supported glyphs. Add the glyphs to the layout marker set or share `BULLET_GLYPHS` so these list items reach markdown intact.</violation>
</file>

_{Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic}

cubic-dev-ai · 2026-08-31T18:25:34Z

 //! Line classification: captions, lists, code detection.

+pub(crate) const BULLET_GLYPHS: &[char] =
+    &['•', '●', '○', '◦', '▪', '▫', '◆', '◇', '■', '□', '‣', '⁃'];


P2: When a PDF places ‣ or ⁃ in standalone marker items, column detection can mistake the marker gutter for a real column because its marker guard does not know these newly supported glyphs. Add the glyphs to the layout marker set or share BULLET_GLYPHS so these list items reach markdown intact.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/markdown/classify.rs, line 4: <comment>When a PDF places `‣` or `⁃` in standalone marker items, column detection can mistake the marker gutter for a real column because its marker guard does not know these newly supported glyphs. Add the glyphs to the layout marker set or share `BULLET_GLYPHS` so these list items reach markdown intact.</comment> <file context> @@ -1,5 +1,15 @@ //! Line classification: captions, lists, code detection. +pub(crate) const BULLET_GLYPHS: &[char] = + &['•', '●', '○', '◦', '▪', '▫', '◆', '◇', '■', '□', '‣', '⁃']; + +fn starts_with_bullet_glyph_and_space(text: &str) -> bool { </file context>

cubic-dev-ai

0 issues found across 2 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic}

Consolidate classify.rs cases per architect spec and add square-bullet merge-items stream-order regression test. Co-authored-by: shahidbeig-a11y <shahidbeig-a11y@users.noreply.github.com>

cubic-dev-ai

All reported issues were addressed across 2 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic}

Add is_standalone_bullet_glyph helper so is_list_marker_column picks up ‣ and ⁃, and test every BULLET_GLYPHS entry in list-item QA. Co-authored-by: shahidbeig-a11y <shahidbeig-a11y@users.noreply.github.com>

cubic-dev-ai

0 issues found across 3 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed

_{Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic}

cubic-dev-ai

1 existing issue remains and 1 new issue found across 5 files

Confidence score: 3/5

src/extractor/layout.rs can create a spurious column for standalone ■, □, ◆, or ◇ markers, scrambling PDF list reading order; preserve the marker/content-gutter rejection guard.
src/markdown/classify.rs may drop standalone newly supported bullet glyphs near charts by classifying them as labels, so align the chart guard with is_standalone_bullet_glyph or the shared glyph set.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/markdown/classify.rs">

<violation number="1" location="src/markdown/classify.rs:6">
P2: When a newly supported bullet glyph is a standalone item near a chart, the chart filter still treats it as a label and drops it. Make the chart guard use `is_standalone_bullet_glyph` or the shared glyph set instead of its stale hard-coded list.</violation>
</file>

_{Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic}

cubic-dev-ai

1 issue found and verified against the latest diff

Confidence score: 4/5

In src/markdown/classify.rs, standalone ▪, ▫, ‣, or ⁃ markers in chart padding can be omitted from the extracted list because their classification does not reach the chart-label filter; reuse is_standalone_bullet_gly... so these markers are retained.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/markdown/classify.rs">

<violation number="1" location="src/markdown/classify.rs:9">
P2: When a standalone `▪`, `▫`, `‣`, or `⁃` lies in chart padding, the new marker classification does not reach the chart-label filter, so the marker can be dropped from the extracted list. Reuse `is_standalone_bullet_glyph` in that filter instead of maintaining the old hard-coded marker list.</violation>
</file>

_{Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic}

Reuse is_standalone_bullet_glyph in chart-adjacent label filtering so standalone ▪▫‣⁃ near charts are not dropped. Keep ◆◇■□ as layout-only gutter markers so list columns still reject spurious valleys. Drop the vacuous heading-sequence assert on a singleton ▪ line. Addresses cubic review on firecrawl#476.

cubic-dev-ai

1 existing issue remains and no new issues found across 3 files (changes from recent commits).

Confidence score: 5/5

src/markdown/mod.rs extends the chart-padding marker guard to ▪, ▫, ‣, and ⁃, but the chart-region test covers only •; loop over the new glyphs in chart_padding_claims_labels_but_not_adjacent_prose to prevent regressions.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 3 unresolved issues from previous reviews.

Re-trigger cubic}

Loop chart_padding_claims_labels_but_not_adjacent_prose over the standalone glyphs the chart-label filter now preserves, so a regression that only keeps • would fail. Addresses cubic review on firecrawl#476.

cubic-dev-ai

All reported issues were addressed across 1 file (changes from recent commits).

_{Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic}

shahidbeig-a11y · 2026-09-23T06:29:09Z

Follow-up: chart-padding guard test now iterates BULLET_GLYPHS end-to-end (0fb69bb) so the pin stays in sync with the shared glyph set.

cubic-dev-ai

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved issue from previous reviews.

Re-trigger cubic}

cursoragent and others added 3 commits August 31, 2026 18:17

test: tighten bullet glyph QA cases for firecrawl#475

0fe1ccf

Add explicit ▪/‣/◆ detection and format tests, middle-dot negative format case, numbered list assertions, and heading rejection test. Co-authored-by: shahidbeig-a11y <shahidbeig-a11y@users.noreply.github.com>

cubic-dev-ai Bot reviewed Aug 31, 2026

View reviewed changes

test: align bullet glyph QA with full BULLET_GLYPHS table

da5d889

Consolidate classify.rs cases per architect spec and add square-bullet merge-items stream-order regression test. Co-authored-by: shahidbeig-a11y <shahidbeig-a11y@users.noreply.github.com>

cubic-dev-ai Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread src/markdown/classify.rs Outdated

fix: share BULLET_GLYPHS with layout marker guard

87d8478

Add is_standalone_bullet_glyph helper so is_list_marker_column picks up ‣ and ⁃, and test every BULLET_GLYPHS entry in list-item QA. Co-authored-by: shahidbeig-a11y <shahidbeig-a11y@users.noreply.github.com>

cubic-dev-ai Bot reviewed Aug 31, 2026

View reviewed changes

shahidbeig-a11y added 3 commits September 21, 2026 14:27

fix(markdown): keep decorative glyph headings out of lists

b997915

fix(markdown): keep decorative glyph headings out of lists

f6d1352

test(layout): cover all list marker glyphs

4a711e3

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/markdown/heading.rs Outdated

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/markdown/classify.rs

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/markdown/classify.rs

Comment thread src/extractor/layout.rs Outdated

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

test(markdown): pin chart-padding guard for ▪▫‣⁃

fe8a147

Loop chart_padding_claims_labels_but_not_adjacent_prose over the standalone glyphs the chart-label filter now preserves, so a regression that only keeps • would fail. Addresses cubic review on firecrawl#476.

cubic-dev-ai Bot reviewed Sep 22, 2026

View reviewed changes

Comment thread src/markdown/mod.rs Outdated

test(markdown): pin all BULLET_GLYPHS in chart-padding guard

0fb69bb

cubic-dev-ai Bot reviewed Sep 23, 2026

View reviewed changes

Conversation

shahidbeig-a11y commented Aug 31, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by cubic

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shahidbeig-a11y commented Sep 23, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shahidbeig-a11y commented Aug 31, 2026 •

edited by cubic-dev-ai Bot

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot Aug 31, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading