Add birdy scrape for batch tweet collection by guzus · Pull Request #46 · guzus/birdy

guzus · 2026-08-20T09:32:14Z

birdy scrape collects tweets from profiles, lists, searches, and tweet URLs in one command.

birdy scrape https://x.com/nasa https://x.com/search?q=moon --max-items 50
birdy scrape --handle elonmusk --handle nasa -n 100
birdy scrape --from nasa --since 2026-01-01 --min-likes 100 --filter media
birdy scrape --search "AI lang:en" --sort both --output csv
birdy scrape --mode replies https://x.com/nasa/status/2090197889947451524

Auto-routes URLs, @handles, tweet IDs, list IDs, and search terms
Compiles --from, --since/--until, --lang, --min-likes, --min-reposts, and --filter into X advanced-search syntax
--sort both runs Latest and Top, then deduplicates
--output json|flat|csv with quote, view, and bookmark counts that bird's JSON contract never emitted
Modes: tweet, search, profile, profile-replies, profile-media, profile-likes, list, replies, quotes, thread, retweeters, favoriters

Existing search, user-tweets, and --json output stay unchanged. Extra metrics stay unexported on the parser type so bird-compatible JSON does not grow new keys.

Review follow-ups (live-tested)

--list now parses list URLs (not just raw ids)
--sort both keeps Latest rows when Top fails (TimelinePrompt)
--handle + a different --from is rejected instead of silently scraping the --from account
--mode likes is no longer an alias for favoriters; use profile-likes or favoriters
Scrape JSON keeps 0 engagement counts
An explicit --sort overrides a search URL f= ranking flag
Web typecheck uses the project's expect helpers so CI can pass

Introduce a unified scrape command that auto-routes mixed tweet, profile, search, and list inputs, compiles structured advanced-search filters, runs Latest+Top search, and exports JSON, flat, or CSV after dedup. Co-authored-by: guzus <guzus@users.noreply.github.com>

guzus

Tested this locally: unit tests, extra planner/collector cases, and live birdy scrape against X with a real account.

What works

Existing tests pass (go test ./internal/scrape ./cmd ./internal/xapi ./internal/birdtool). Live happy paths also work:

--handle nasa -n 3 IDs match birdy user-tweets nasa -n 3 --json exactly
--search from:nasa IDs match birdy search --json
Extra metrics (quoteCount / viewCount / bookmarkCount / lang) show up on scrape rows and stay off native --json (read / user-tweets / search)
--output flat, --mode thread|replies|quotes|retweeters, positional list URLs, @nasa vs bare AI, structured --from/--since/--min-likes/--filter, and --max-items-per-target all behaved
Validation (no targets, unknown mode/output/filter, bad date, -n 10001) fails before hitting X

Must-fix (reproduced live)

--list <URL> is broken. Flag help says "id or URL". --list https://x.com/i/lists/1748648376080666720 sends the raw URL as listId and X returns unrecognized timeline response shape. The same URL as a positional works.
--sort both drops data. --sort latest and --sort top both succeed for AI lang:en. --sort both hits TimelinePrompt on the second search and prints nothing, because collectScrapeRows continues on error even when searchJob returned the Latest tweets. A collector unit test that fails Top on the second call gets rows=[].
--handle nasa --from elonmusk scrapes Elon, not NASA. compiled already contains from:elonmusk, so the handle is discarded.

Should-fix

--mode likes aliases to favoriters. --mode likes --handle nasa errors with mode favoriters requires tweet ids.
Scrape JSON omitemptys zero counts. Same NASA tweets: user-tweets has likeCount: [2190, 0, 0], scrape omits the zeros. CSV still prints 0.
Search URL f=live overrides --sort top.
README example tweet 1846987139428634858 is dead (TweetDetail response contained no usable tweets). Use a live id.
--mode profile-likes --handle nasa (hidden likes) returns an opaque GraphQL shape error instead of a clear "likes unavailable" message.

CI

The red test job is not from scrape. make verify dies in web typecheck on App.test.tsx (toHaveLength / .not), which landed on main in "Make X URLs clickable in birdy web". Other tests in that file use .toBe(false). This branch cannot go green until that is fixed on main or here.

The three live functional bugs and the zero-count JSON contract should be fixed before merge. Happy to re-test once those are addressed.

guzus · 2026-08-20T10:05:33Z

+	for _, id := range r.ListIDs {
+		id = strings.TrimSpace(strings.TrimPrefix(id, "list:"))
+		if id == "" {
+			return nil, fmt.Errorf("empty list id")
+		}
+		targets = append(targets, Target{Kind: KindList, Value: id, Source: id})
+	}


--list is documented as accepting a URL, but this path only strips a list: prefix. Live:

birdy scrape --list https://x.com/i/lists/1748648376080666720 -n 2 # Error: x api: unrecognized timeline response shape: no known instructions root

The same URL as a positional works, because Classify extracts the numeric id. Run --list values through Classify (or at least the list-URL regex) so the advertised flag works.

guzus · 2026-08-20T10:05:34Z

+		case scrape.KindSearch:
+			tweets, err := searchJob(ctx, client, job)
+			if err != nil {
+				runErr = err
+				continue
+			}
+			for _, t := range tweets {
+				addTweet(t, job)
+			}


searchJob(..., SortBoth) already returns Latest tweets plus the Top error, but this continue drops those tweets. Live: --sort latest and --sort top both succeed for AI lang:en, while --sort both exits 1 with empty stdout (TimelinePrompt on the second search).

A unit test that fails Top on the second SearchWith call gets rows=[]. Add the Latest tweets before recording runErr, otherwise the README's --sort both example loses data on a flaky Top page.

guzus · 2026-08-20T10:05:36Z

+	case KindProfile:
+		if filters.HasConstraints() || compiled != "" {
+			query := compiled
+			if !strings.Contains(query, "from:") {
+				query = strings.TrimSpace("from:" + target.Value + " " + query)
+			}
+			job.Kind = KindSearch
+			job.Query = strings.TrimSpace(query)
+		}


If compiled already contains from:, the handle is dropped. Live:

birdy scrape --handle nasa --from elonmusk -n 3 # authors: elonmusk, elonmusk, elonmusk

HasConstraints also ignores From, so this only becomes a search because Compile() emitted from:elonmusk. Either reject --handle + --from as conflicting, or keep from:<handle> AND from:<from> (or document that --from replaces the handle).

guzus · 2026-08-20T10:05:36Z

+	case ModeFavoriters, "likes":
+		return ModeFavoriters, nil


likes is a surprising alias for favoriters. Live:

birdy scrape --mode likes --handle nasa # Error: mode favoriters requires tweet ids or URLs, got nasa

Users will read this as profile likes (--mode profile-likes / https://x.com/nasa/likes). Drop the likes alias or map it to profile-likes when the target is a handle.

guzus · 2026-08-20T10:05:37Z

+	LikeCount      int      `json:"likeCount,omitempty"`
+	RetweetCount   int      `json:"retweetCount,omitempty"`
+	ReplyCount     int      `json:"replyCount,omitempty"`
+	QuoteCount     int      `json:"quoteCount,omitempty"`
+	ViewCount      int      `json:"viewCount,omitempty"`
+	BookmarkCount  int      `json:"bookmarkCount,omitempty"`


omitempty on counts makes zero likes disappear from JSON. Live, the same 3 NASA tweets:

birdy user-tweets nasa -n 3 --json → likeCount: [2190, 0, 0]

birdy scrape --handle nasa -n 3 → [2190, <omitted>, <omitted>]

CSV still prints 0. Native/bird JSON keeps zeros on purpose (TestNativeTweetJSONMatchesBird). Scrape consumers cannot tell "0 likes" from "metric missing". Drop omitempty on the numeric engagement fields.

guzus · 2026-08-20T10:05:38Z

+		Value:  target.Value,
+		Source: target.Source,
+		Limit:  limit,
+		Sort:   firstNonEmpty(target.Sort, sort),


firstNonEmpty(target.Sort, sort) lets a search URL's f=live override --sort. Plan({Positionals: ["https://x.com/search?q=moon&f=live"], Sort: "top"}) yields Sort: latest. If the user passed --sort explicitly it should win; URL sort should only fill the default.

Parse --list URLs to numeric ids, keep Latest rows when Top search fails, reject a conflicting --handle/--from pair, stop aliasing --mode likes to favoriters, keep zero engagement counts in JSON, and let an explicit --sort override a search URL ranking flag. Co-authored-by: guzus <guzus@users.noreply.github.com>

Use the project's expect helpers so bun typecheck accepts the target=_blank and no-link assertions. Co-authored-by: guzus <guzus@users.noreply.github.com>

guzus commented Aug 20, 2026

View reviewed changes

cursoragent and others added 2 commits August 20, 2026 10:57

Fix web typecheck for clickable X URL tests

48cc020

Use the project's expect helpers so bun typecheck accepts the target=_blank and no-link assertions. Co-authored-by: guzus <guzus@users.noreply.github.com>

guzus marked this pull request as ready for review August 20, 2026 11:12

guzus merged commit 4f3f6c3 into main Aug 20, 2026
1 check passed

Navigation Menu

Repository navigation

Add birdy scrape for batch tweet collection - #46

Add birdy scrape for batch tweet collection#46
guzus merged 3 commits into
mainfrom
cursor/tweet-scrape-df46

guzus commented Aug 20, 2026 •

edited by cursor Bot

Loading

Uh oh!

guzus left a comment

Uh oh!

guzus Aug 20, 2026

Uh oh!

guzus Aug 20, 2026

Uh oh!

guzus Aug 20, 2026

Uh oh!

guzus Aug 20, 2026

Uh oh!

guzus Aug 20, 2026

Uh oh!

guzus Aug 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

guzus commented Aug 20, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review follow-ups (live-tested)

Uh oh!

guzus left a comment

Choose a reason for hiding this comment

What works

Must-fix (reproduced live)

Should-fix

CI

Uh oh!

guzus Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

guzus Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

guzus Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

guzus Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

guzus Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

guzus Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

guzus commented Aug 20, 2026 •

edited by cursor Bot

Loading