[heraldb]

Problem

On a mailbox with ~29,000 messages in INBOX, Rav only ever showed 76 of them.
Other large folders (Sent, 23,650) loaded fine.

On first load, list_messages fetches the ~100 newest UIDs, then spawns one
background UID FETCH 1:<n> for all older headers. If that single fetch
fails, the folder stays incomplete for good:

• the failure is only logged (Background sync: failed (will retry on next request)),
  but the retry never happens: the cache is no longer empty, so later
  requests take the "stale" path;
• the sync worker (CONDSTORE and full paths) only fetches UIDs above the
  highest cached UID, so older gaps are never filled.

The trigger here was one message whose BODYSTRUCTURE has an attachment
filename sent as a literal with a raw 8-bit byte (Latin-1 ®, 0xAE):

("attachment" ("filename" {28}\r\nFiber Foods - PrimeJack\xAE.pdf))

The IMAP parser rejects it (TakeWhile1), the connection drops, and the
whole 120k-UID fetch is lost. A second folder (Junk) failed the same one-shot
fetch on database is locked, and was never retried either.

Change

Gap-filling backfill in the sync worker (realtime/sync.rs)

• After the CONDSTORE/full tiers, if a folder that has been loaded before
  (cached > 0) has fewer cached messages than the server's EXISTS, the
  worker fetches the UID list, computes the missing UIDs, and fetches their
  headers newest first in batches of 500, storing each batch as it
  arrives.
• At most 5,000 messages per folder per cycle, so a large first sync doesn't
  delay new mail in other folders; the remainder follows on later cycles.
• Never-opened folders (nothing cached) are not downloaded in the background.

Resilient to unparseable messages

• A failed batch is bisected. A single UID that still fails is fetched with
  the new ImapClient::fetch_headers_basic: the same items minus
  BODYSTRUCTURE, so the message is listed and opens normally, just without
  the attachment flag. If that fails too, the UID is skipped with a warning.
• The backfill stops for the cycle after 20 consecutive failures, since the
  server is then likely unreachable rather than one message being bad.

Cold start (routes/messages/mod.rs)

• Older headers are now handed to the worker's backfill instead of the
  separate one-shot task (sync_remaining_headers removed).
• If the recent-headers fetch itself fails, it retries with
  fetch_headers_basic, so a bad message among the newest 100 no longer
  makes the folder impossible to open.

Refactoring

• RealImapClient::fetch_headers_with_items holds the shared fetch/parse
  code for fetch_headers and fetch_headers_basic.
• upsert_headers/index_headers helpers in sync.rs replace the inline
  copy in fetch_and_store_new.

Tests

• sync_backfills_older_messages_missing_from_cache: a folder with only its
  newest message cached and unchanged MODSEQ gets all 1,200 older messages
  over three batches, including one whose full header fetch always fails,
  which is stored without the attachment flag. Verified to fail with the gap
  fill disabled.
• sync_does_not_backfill_never_loaded_folders
• to_uid_set_collapses_consecutive_runs
• MockImapClient now honours the requested UID set, and can simulate
  unparseable UIDs (with_unparseable_uids).
• cargo test: all pass.

Verified against the affected 29k-message mailbox (Dovecot 2.3): the INBOX
filled in completely within a few minutes, and the problem message is listed.

Known limitation

A message that can't be fetched even without BODYSTRUCTURE stays missing,
and is retried (with a warning) on each sync cycle.

Note: CI clippy currently fails on main with Rust 1.99 for unrelated
reasons; that's fixed separately in fix/clippy-rust-1.99.

🤖 Generated with Claude Code