The application is split into three layers:
Rails APIIngestion + background executionReact frontend
The Rails app is the system of record. Every crawl stores a raw source snapshot first, then parses and normalizes into shared opportunity records.
- React calls
GET /opportunitiesorGET /opportunities/:id. - Rails controllers delegate filtering to
Search::OpportunitySearch. - Serializers shape API responses for list and detail views.
POST /sources/:id/syncenqueuesSourceSyncJob.SourceSyncJobdelegates toSourceIngestion::SyncSource.- The ingestion service resolves a crawler via
Crawlers::Registry. - Raw payloads are persisted as
SourceRecordrows with fingerprints. - Parsed attributes flow through
Normalization::OpportunityNormalizer. - Buyers and opportunities are upserted into normalized tables.
Every fetch persists the original payload before normalization. That gives us:
- replayability when parsers change
- auditability for source changes
- deterministic reprocessing
Search::*owns filtering and ranking.Normalization::*owns schema mapping.SourceIngestion::*owns persistence and orchestration.Crawlers::*owns source-specific fetch and parse logic.Ai::*owns narrow enrichment behavior.
Sidekiq is the queue engine for:
- scheduled crawls
- retries
- enrichment work
- future dedupe and change detection
- Add persistent
watchlistsandsaved_searchestables and endpoints. - Expand enrichment jobs to call a configurable LLM provider.
- Add
opportunity_updates,contacts, andvendors. - Add document storage and extraction for attachments.
- Add semantic retrieval with
pgvectorwhen document search justifies it.