Distributed Resource Query Engine

A distributed, shard-based metadata query engine built with a Python orchestration layer and horizontally scaled retrieval nodes. The system demonstrates distributed coordination, connection pooling, tail latency mitigation, and policy-aware filtering under load.

---

🚀 Architecture Overview

Client
   ↓
Python Orchestrator (FastAPI)
   ↓
-----------------------------------------
| node1 | node2 | node3 | node4 | node5 |
-----------------------------------------
Shard-based Retrieval Services

---

🧩 Components

Orchestrator (Coordinator)

• Async fan-out to all shards
• Timeout-based shard coordination
• Partial result handling
• Policy-aware filtering
• Global deterministic sorting
• Pagination
• Per-shard latency tracking
• Connection pooling

Retrieval Nodes (Shards)

• Independent FastAPI services
• Each loads a different dataset
• Performs local search on metadata
• Exposes /search and /health

---

🧠 Key Design Principles

1. Sharding

Data is partitioned across 5 independent retrieval nodes.

This simulates:

• Horizontal scaling
• Distributed metadata indexing
• Independent shard responsibility

---

2. Async Fan-Out Coordination

The orchestrator queries all shards concurrently:

asyncio.wait(tasks, timeout=SHARD_TIMEOUT_S)

This ensures:

• Parallel shard execution
• Reduced query latency
• Non-blocking coordination

---

3. Tail Latency Mitigation

Instead of waiting indefinitely for slow shards:

• Coordinator waits up to 300ms
• Cancels slow shard requests
• Returns partial results

Response includes:

{
  "partial": true,
  "timed_out_nodes": [....]
}

This models real distributed search engines.

---

4. Connection Pooling

A global httpx.AsyncClient is initialized at application startup:

httpx.AsyncClient(
    limits=httpx.Limits(
        max_connections=2000,
        max_keepalive_connections=1000
    )
)

This enables:

• TCP connection reuse
• Reduced handshake overhead
• Controlled resource usage
• High-QPS scalability

---

5. Policy-Aware Filtering

Documents are filtered based on:

• Visibility
• Allowed roles

doc["visibility"] == "public"
and role in doc["allowed_roles"]

This simulates enterprise access control enforcement at the coordination layer.

---

6. Deterministic Global Sorting

All merged shard results are sorted globally before pagination.

Prevents:

• Inconsistent pagination
• Order instability
• Non-deterministic page results

---

📊 Performance Model

Concurrency is calculated using Little’s Law:

Concurrency = QPS × Latency

Example:

• 3,000 QPS
• 5 shards
• 120ms shard latency

Outbound calls per second:

3,000 × 5 = 15,000

Concurrent shard calls:

15,000 × 0.12 = 1,800

Connection pool limits are configured accordingly.

---

🛠 Features Implemented

• Distributed shard-based retrieval
• Async orchestration
• Shard timeout budget
• Partial response signaling
• Per-shard latency measurement
• Deterministic sorting
• Pagination
• Connection pooling
• Resource lifecycle management

---

🐳 Running the Project

docker compose up --build

Access:

http://localhost:9000/query?q=ai

---

🧪 Example Response

{
  "total_results": 12,
  "page": 1,
  "limit": 5,
  "partial": false,
  "timed_out_nodes": [],
  "shard_latencies_ms": {
    "http://node1:8000/search": 112.4
  },
  "results": [...]
}

---

🔍 Engineering Tradeoffs

Choice	Reason
Timeout-based partial results	Availability over completeness
Global sorting before pagination	Deterministic user experience
Connection pooling	High throughput and resource efficiency
Centralized policy filtering	Consistent enforcement

---

📈 Future Enhancements

• C++ retrieval module implementation
• Inverted indexing instead of linear scan
• Prometheus metrics export
• Circuit breaker pattern
• Hedged requests (replica querying)
• Load testing validation
• Distributed tracing support

---

🎯 What This Project Demonstrates

• Distributed coordination patterns
• Tail latency mitigation strategies
• Connection pooling & lifecycle management
• Horizontal sharding architecture
• Production-oriented API design
• Performance modeling using concurrency math