Each new source should fit the same ingestion pattern:
- fetch raw payloads
- snapshot them into
source_records - parse source-specific fields
- normalize into the shared schema
- enrich and dedupe later
For a new source example_source, add:
lib/crawlers/example_source/
crawler.rb
client.rb
parser.rb
Then register it in lib/crawlers/registry.rb.
- prefer official APIs over scraping
- respect robots, terms, and rate limits
- return raw payload objects only
- map source-specific keys into a stable intermediate hash
- do not normalize enums here
- keep enough raw text for search and auditability
- map parser output into the shared schema
- normalize enums like status and set-aside
- preserve source URLs and identifiers
- Persist raw payloads before normalization.
- Keep crawlers idempotent.
- Always produce a fingerprint per payload.
- Capture enough raw content to support parser replays.
- Avoid source-specific behavior leaking into the API layer.
- Keep API keys and secrets in environment variables or a secret manager, not in
source.settingsseeds or committed config files.