Skip to content

docs(crawlee): warn against router: as top-level CheerioCrawler option#2735

Draft
DaveHanns wants to merge 1 commit into
apify:masterfrom
DaveHanns:fix/f76-cheerio-router-anti-pattern
Draft

docs(crawlee): warn against router: as top-level CheerioCrawler option#2735
DaveHanns wants to merge 1 commit into
apify:masterfrom
DaveHanns:fix/f76-cheerio-router-anti-pattern

Conversation

@DaveHanns

Copy link
Copy Markdown
Contributor

Summary

When reviewing AI-generated Crawlee code (from ChatGPT, Cursor, Claude Code, etc.), one very common mistake is passing the router under a router option — or in addition to requestHandler:

// This does not work:
new CheerioCrawler({ router, requestHandler: async (ctx) => { /* ... */ } });

Crawlee has no top-level router option. The initializing_and_setting_up.md lesson in scraping_basics_legacy/challenge/ already shows the correct pattern (requestHandler: router), so this PR adds a short :::caution admonition immediately after that code sample which:

  1. States the rule (router is passed as requestHandler, there is no router option).
  2. Quotes the two error messages a reader will see verbatim (runtime ow error + TS2353), so search hits from those exact strings land here.
  3. Shows the correct + incorrect snippet side-by-side.
  4. Notes that this is a recurring AI-suggested pattern to watch for.

No existing code paths or examples change; the addition is a single admonition block.

Evidence

Both error messages are reproducible with any recent Crawlee release.

Runtime (ow predicate rejection at construction):

ArgumentError: Did not expect property `router` to exist, got ...
    at new HttpCrawler ...

TypeScript compile-time:

error TS2353: Object literal may only specify known properties,
and 'router' does not exist in type 'CheerioCrawlerOptions'.

Minimal reproducer:

import { CheerioCrawler, createCheerioRouter } from 'crawlee';

const router = createCheerioRouter();
router.addDefaultHandler(async ({ log }) => log.info('hi'));

// ❌ fails: no top-level `router` option
new CheerioCrawler({ router, requestHandler: async () => {} });

// ✅ works
new CheerioCrawler({ requestHandler: router });

Test plan

  • Docusaurus renders the new :::caution block correctly in light + dark themes.
  • The two text code blocks inside the admonition render without syntax-highlighter warnings.
  • Anchors / TOC of the page are unaffected (no new heading was added).

Surfaced during an evaluation of Apify surfaces for agent-driven Actor development.

Reviewing AI-generated Crawlee code, one recurring mistake is passing the
router under a `router` option (or in addition to `requestHandler`):

    new CheerioCrawler({ router, requestHandler: async (ctx) => { ... } })

Crawlee has no top-level `router` option. At runtime this fails ow validation
with `ArgumentError: Did not expect property 'router' to exist ... at new
HttpCrawler`. In TypeScript projects it fails at compile time with
`TS2353: 'router' does not exist in type 'CheerioCrawlerOptions'`.

The scraping-basics-legacy "Initialization and setting up" lesson already
shows the correct pattern (`requestHandler: router`) — this commit adds a
short caution admonition right below that snippet that spells out the
anti-pattern, quotes both error messages verbatim, and calls out that this
is a common suggestion from AI chat tools and agentic coding assistants.

No code paths or examples change; the addition is a single admonition block.

Surfaced during an evaluation of Apify surfaces for agent-driven Actor development.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@DaveHanns DaveHanns marked this pull request as draft July 5, 2026 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants