Skip to content

fix: bound is_chart_bar_cluster's O(n^3) cost on dense rect clusters - #506

Open
MADENIYOU wants to merge 1 commit into
firecrawl:mainfrom
MADENIYOU:fix/bound-chart-cluster-cost
Open

MADENIYOU wants to merge 1 commit into
firecrawl:mainfrom
MADENIYOU:fix/bound-chart-cluster-cost

Conversation

@MADENIYOU

@MADENIYOU MADENIYOU commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Split out from #221 per @abimaelmartell's review there — the chart-cluster performance work was originally bundled into that PR, but a later refinement of it caused a real regression (a 20-column statistical table getting malformed into 9 columns), so it was dropped from #221 to keep that PR focused on the decompression-bomb fix. This PR reintroduces the underlying performance fix, this time with a deliberately simpler, safer design.

The problem

bar_family (used by is_chart_bar_cluster to distinguish bar charts from cell-rect grids) iterates every rect in a cluster as an anchor, rebuilds a same-breadth "family" per anchor, then runs a nested any/filter over that family plus a per-member full-page item scan — O(n³) in the cluster size in the worst case.

tests/fixtures/bits_pilani_feedback.pdf (a dense feedback-form PDF) clusters into ~2000-2008 touching rects via cluster_rects. On current main (no cap at all — this fix was never independently merged), this single file takes:

time ./target/release/pdf2md tests/fixtures/bits_pilani_feedback.pdf
# 31.5s

The fix

Added MAX_CHART_CLUSTER_RECTS (500) as an early bailout in is_chart_bar_cluster: clusters above that size are treated as "not chart-like" without running the expensive checks at all, deferring to the normal table-grid detectors — which is what a genuinely dense table/form cluster this size actually is anyway. 500 is an order of magnitude below the pathological range measured in the profiled fixture (~2000 rects), leaving real headroom above plausible legitimate chart sizes.

After the fix, the same file:

time ./target/release/pdf2md tests/fixtures/bits_pilani_feedback.pdf
# 3.6s (~8.7x)

Output is byte-identical to before, apart from the reported processing time in the CLI banner — confirmed via diff.

Why not the bucket-sampling version from before

The original attempt on #221 went further: instead of unconditionally excluding oversized clusters, it bucketed/subsampled them by position to still recognize genuinely huge multi-series bar charts (avoiding routing them into table detection as garbage tables). That was correctly flagged in review as causing a real regression — a legitimate large table's rect cluster could get misclassified as chart-like under the biased sample, silently dropping columns (a demonstrated 20-column → 9-column malformation on a real statistical table).

Given the asymmetry — "a huge chart renders as a mediocre table" (the unconditional-cutoff failure mode) is a known, bounded, cosmetic issue, while "a legitimate table silently loses columns" (the subsampling failure mode) is real data loss — this PR intentionally does not reintroduce the subsampling refinement. The constant's doc comment records this tradeoff explicitly for whoever wants to revisit detecting genuinely oversized charts, ideally validated against a real large-table regression fixture in the eval set first.

Testing

  • New test is_chart_bar_cluster_bails_out_above_max_size: a genuinely chart-shaped cluster (spaced bars, increasing heights, numeric labels — not just "many rects") just above the cap resolves to false.
  • New sibling test is_chart_bar_cluster_detects_genuine_chart_below_the_size_cap: the same shape just below the cap still resolves to true — proves the oversized test's false comes specifically from the size bailout, not from the shape failing has_chart_bar_signature on its own merits.
  • No wall-clock assertion in either test (flagged as CI-flaky in an earlier review round on this same fix) — the real performance win is measured manually above.
  • cargo fmt, cargo test (176 integration + 1165 unit + 2 doc, all passing, 3 new), cargo clippy --all-targets -- -D warnings — verified the remaining clippy output is byte-identical (same files, same errors) to a clean main checkout run side by side; this PR introduces zero new findings.

🤖 Generated with Claude Code


Summary by cubic

Bounding is_chart_bar_cluster's cubic cost on dense rect clusters so PDFs with thousands of touching rects no longer stall. A dense feedback-form fixture dropped from ~31s to ~3.6s with unchanged output.

  • Adds MAX_CHART_CLUSTER_RECTS (500); clusters above it skip the expensive checks and defer to the table-grid detectors.
  • The earlier bucket-sampling approach stays out — it risked silently dropping columns on legitimate large tables, while the unconditional cutoff only makes huge charts render as mediocre tables.
  • Three new tests verify a genuine chart below the cap still classifies chart-like and the oversized bailout fires on size alone.

Written for commit baeb940. Summary will update on new commits.

Review in cubic

bar_family (used by is_chart_bar_cluster to distinguish bar charts from
cell-rect grids) iterates every rect as an anchor, rebuilds a
same-breadth "family" per anchor, then runs a nested any/filter over
that family plus a per-member full-page item scan — O(n^3) in the
cluster size in the worst case.

tests/fixtures/bits_pilani_feedback.pdf (a dense feedback-form PDF)
clusters into ~2000-2008 touching rects via cluster_rects, driving
is_chart_bar_cluster's cost to 30+ seconds for this single file on
current main (verified: 31.5s before this fix, 3.6s after — output
byte-identical apart from the reported processing time).

No real bar chart has thousands of bars, so this adds
MAX_CHART_CLUSTER_RECTS (500) as an early bailout — clusters above
that size are treated as "not chart-like" instead of running the
expensive checks, deferring to the normal table-grid detectors (which
is what a genuinely dense table/form cluster this size actually is).
500 is an order of magnitude below the pathological range measured in
the profiled fixture, leaving real headroom above plausible legitimate
chart sizes.

This reintroduces a simpler, safer version of a fix that was
originally bundled into firecrawl#221 and dropped from that PR after review
found a regression: a later "bucket/subsample oversized clusters
instead of unconditionally excluding them" refinement (meant to still
recognize genuinely huge multi-series charts) was shown, against a
real 20-column statistical table, to occasionally misclassify a
legitimate large table's rect cluster as chart-like under the biased
sample, silently dropping table columns. That refinement is
deliberately NOT reintroduced here — an unconditional exclusion above
the cap is a known, bounded regression (a genuinely huge chart renders
as a mediocre table) rather than a data-loss risk on real tables. The
constant's doc comment records this tradeoff for whoever revisits it.

New tests: is_chart_bar_cluster_bails_out_above_max_size (an oversized
but genuinely chart-shaped cluster resolves to `false` purely from the
size bailout) and its sibling
is_chart_bar_cluster_detects_genuine_chart_below_the_size_cap (the same
shape below the cap still resolves to `true`, proving the oversized
test's `false` isn't just the shape failing has_chart_bar_signature on
its own merits). No wall-clock assertion — flagged as CI-flaky in an
earlier review round; the real performance win is measured manually
and documented above.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Shadow auto-approve: would auto-approve. Adds a 500-rect cap to is_chart_bar_cluster to fix an O(n^3) stall (31s→3.6s) with byte-identical output. Tests pin the corrected behavior around the cap; the large-chart tradeoff is documented.

Re-trigger cubic

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant