Conversation
bar_family (used by is_chart_bar_cluster to distinguish bar charts from cell-rect grids) iterates every rect as an anchor, rebuilds a same-breadth "family" per anchor, then runs a nested any/filter over that family plus a per-member full-page item scan — O(n^3) in the cluster size in the worst case. tests/fixtures/bits_pilani_feedback.pdf (a dense feedback-form PDF) clusters into ~2000-2008 touching rects via cluster_rects, driving is_chart_bar_cluster's cost to 30+ seconds for this single file on current main (verified: 31.5s before this fix, 3.6s after — output byte-identical apart from the reported processing time). No real bar chart has thousands of bars, so this adds MAX_CHART_CLUSTER_RECTS (500) as an early bailout — clusters above that size are treated as "not chart-like" instead of running the expensive checks, deferring to the normal table-grid detectors (which is what a genuinely dense table/form cluster this size actually is). 500 is an order of magnitude below the pathological range measured in the profiled fixture, leaving real headroom above plausible legitimate chart sizes. This reintroduces a simpler, safer version of a fix that was originally bundled into firecrawl#221 and dropped from that PR after review found a regression: a later "bucket/subsample oversized clusters instead of unconditionally excluding them" refinement (meant to still recognize genuinely huge multi-series charts) was shown, against a real 20-column statistical table, to occasionally misclassify a legitimate large table's rect cluster as chart-like under the biased sample, silently dropping table columns. That refinement is deliberately NOT reintroduced here — an unconditional exclusion above the cap is a known, bounded regression (a genuinely huge chart renders as a mediocre table) rather than a data-loss risk on real tables. The constant's doc comment records this tradeoff for whoever revisits it. New tests: is_chart_bar_cluster_bails_out_above_max_size (an oversized but genuinely chart-shaped cluster resolves to `false` purely from the size bailout) and its sibling is_chart_bar_cluster_detects_genuine_chart_below_the_size_cap (the same shape below the cap still resolves to `true`, proving the oversized test's `false` isn't just the shape failing has_chart_bar_signature on its own merits). No wall-clock assertion — flagged as CI-flaky in an earlier review round; the real performance win is measured manually and documented above. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
No issues found across 1 file
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would auto-approve. Adds a 500-rect cap to is_chart_bar_cluster to fix an O(n^3) stall (31s→3.6s) with byte-identical output. Tests pin the corrected behavior around the cap; the large-chart tradeoff is documented.
Re-trigger cubic
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out from #221 per @abimaelmartell's review there — the chart-cluster performance work was originally bundled into that PR, but a later refinement of it caused a real regression (a 20-column statistical table getting malformed into 9 columns), so it was dropped from #221 to keep that PR focused on the decompression-bomb fix. This PR reintroduces the underlying performance fix, this time with a deliberately simpler, safer design.
The problem
bar_family(used byis_chart_bar_clusterto distinguish bar charts from cell-rect grids) iterates every rect in a cluster as an anchor, rebuilds a same-breadth "family" per anchor, then runs a nested any/filter over that family plus a per-member full-page item scan — O(n³) in the cluster size in the worst case.tests/fixtures/bits_pilani_feedback.pdf(a dense feedback-form PDF) clusters into ~2000-2008 touching rects viacluster_rects. On currentmain(no cap at all — this fix was never independently merged), this single file takes:The fix
Added
MAX_CHART_CLUSTER_RECTS(500) as an early bailout inis_chart_bar_cluster: clusters above that size are treated as "not chart-like" without running the expensive checks at all, deferring to the normal table-grid detectors — which is what a genuinely dense table/form cluster this size actually is anyway. 500 is an order of magnitude below the pathological range measured in the profiled fixture (~2000 rects), leaving real headroom above plausible legitimate chart sizes.After the fix, the same file:
Output is byte-identical to before, apart from the reported processing time in the CLI banner — confirmed via
diff.Why not the bucket-sampling version from before
The original attempt on #221 went further: instead of unconditionally excluding oversized clusters, it bucketed/subsampled them by position to still recognize genuinely huge multi-series bar charts (avoiding routing them into table detection as garbage tables). That was correctly flagged in review as causing a real regression — a legitimate large table's rect cluster could get misclassified as chart-like under the biased sample, silently dropping columns (a demonstrated 20-column → 9-column malformation on a real statistical table).
Given the asymmetry — "a huge chart renders as a mediocre table" (the unconditional-cutoff failure mode) is a known, bounded, cosmetic issue, while "a legitimate table silently loses columns" (the subsampling failure mode) is real data loss — this PR intentionally does not reintroduce the subsampling refinement. The constant's doc comment records this tradeoff explicitly for whoever wants to revisit detecting genuinely oversized charts, ideally validated against a real large-table regression fixture in the eval set first.
Testing
is_chart_bar_cluster_bails_out_above_max_size: a genuinely chart-shaped cluster (spaced bars, increasing heights, numeric labels — not just "many rects") just above the cap resolves tofalse.is_chart_bar_cluster_detects_genuine_chart_below_the_size_cap: the same shape just below the cap still resolves totrue— proves the oversized test'sfalsecomes specifically from the size bailout, not from the shape failinghas_chart_bar_signatureon its own merits.cargo fmt,cargo test(176 integration + 1165 unit + 2 doc, all passing, 3 new),cargo clippy --all-targets -- -D warnings— verified the remaining clippy output is byte-identical (same files, same errors) to a cleanmaincheckout run side by side; this PR introduces zero new findings.🤖 Generated with Claude Code
Summary by cubic
Bounding
is_chart_bar_cluster's cubic cost on dense rect clusters so PDFs with thousands of touching rects no longer stall. A dense feedback-form fixture dropped from ~31s to ~3.6s with unchanged output.MAX_CHART_CLUSTER_RECTS(500); clusters above it skip the expensive checks and defer to the table-grid detectors.Written for commit baeb940. Summary will update on new commits.