Repository navigation
Charge the Lightpanda headless browser like a training crawler - #39
Merged
Merged
Conversation
…crawler Lightpanda/1.0 was 86% of p0dcasters traffic on 2026-10-06 (21,263 of 24,787 requests in an hour) from 999 residential addresses. Each one loads a page plus its prefetches (~81 requests in 27 s) and leaves, so it never crosses the 100/min per-address allowance. It runs JavaScript, so every visit showed up as a live reader in analytics (322 "active users"). It names itself, so it goes on the TRAINING list: 402 with the crawl-pass offer, and robots.txt now names it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The site showed 322 "active users". In the dev2 nginx log for the last hour, 21,263 of 24,787 requests (86%) came from
Lightpanda/1.0, spread over 999 residential IPs. Each IP loads one page, fires Next's link prefetches (~81 requests in ~27 s), and leaves. That never trips the 100/min per-address throttle. Because Lightpanda runs JS, every one of those visits counted as a live visitor in analytics.What
src/lib/crawl-gateway.ts: adds"Lightpanda"toTRAINING, so it gets a 402 with the crawl-pass offer, the same as GPTBot. robots.txt now names it as well.test/crawl-gateway.test.mjs: Lightpanda gets 402 on page loads and on RSC prefetches, a real Chrome request passes through, and robots.txt stays readable. The two Lightpanda tests fail without the change.bun test: 36 pass.tsc --noEmit: clean.🤖 Generated with Claude Code