Skip to content

debugging.md

Latest commit

 

History

History
211 lines (155 loc) · 6.89 KB

debugging.md

File metadata and controls

211 lines (155 loc) · 6.89 KB
Raw

Debugging Democritus

This guide covers logging, diagnostics, state files, and workflows for debugging the Democritus daemon.

Log Levels

Democritus uses the tracing crate for structured logging. Control log output with the RUST_LOG environment variable.

Setting Log Levels

# Default (info level)
./target/release/daemon daemon --config config.toml

# Debug — shows routing decisions, peer connections, decomposition details
RUST_LOG=debug ./target/release/daemon daemon --config config.toml

# Trace — extremely verbose, shows every packet and internal state change
RUST_LOG=trace ./target/release/daemon daemon --config config.toml

# Module-specific logging
RUST_LOG=daemon::p2p=debug,daemon::coordinator=trace ./target/release/daemon daemon --config config.toml

Windows (PowerShell):

$env:RUST_LOG = "debug"
.\target\release\daemon.exe daemon --config config.toml

Useful Module Filters

Filter What it shows
daemon::p2p=debug Peer connections, DHT queries, Gossipsub events
daemon::coordinator=debug Task routing, expert selection, MoA aggregation
daemon::decomposition=debug Intent classification, complexity scoring, task DAG
daemon::inference=debug Model loading, token generation, context management
daemon::reputation=debug Trust score changes, PoI audit results
daemon::embedding=debug Vector generation, LSH quantization, cosine similarity
daemon::api=debug REST request/response handling
daemon::grpc=debug gRPC service calls
daemon::ws=debug WebSocket connections and messages
daemon::ledger=debug Credit transactions, balance changes
daemon::sanitizer=debug Input/output sanitization decisions
libp2p=debug Low-level libp2p networking events

Log Output Files

When running as a service or sidecar, logs go to:

  • daemon_stdout.log — Standard output (info/debug/trace messages)
  • daemon_stderr.log — Standard error (warnings and errors)

Health Diagnostics

REST Health Endpoint

curl http://127.0.0.1:8080/v1/health | python -m json.tool

Response fields:

Field Meaning
overall_healthy true if all subsystems are operational
p2p_status.routing_table_size Number of peers in the Kademlia routing table
p2p_status.warm_connections_count Number of active QUIC connections
p2p_status.private_swarm_enabled Whether PSK is configured
ledger_status.ledger_entries_count Number of tracked peer accounts
ledger_status.credit_ledger_healthy Whether the credit system is operational
hardware_status.gpu_detected Whether a GPU was found
hardware_status.vram_total_bytes Total GPU VRAM (0 if no GPU)
hardware_status.system_memory_total_bytes Total system RAM

CLI Health Probe

./target/release/daemon daemon --config config.toml --health

Runs a deep diagnostic probe and outputs a detailed report covering P2P, ledger, hardware, and model status.

State Files

The state_dir (default: ./state) contains persistent state files:

reputation.json

Stores the per-peer reputation table:

{
    "12D3KooW...": {
        "peer_id": "12D3KooW...",
        "successful_inferences": 42,
        "failed_inferences": 2,
        "poi_pass": 10,
        "poi_fail": 0,
        "last_seen": "2026-06-01T12:00:00Z",
        "avg_tokens_per_sec": 15.3,
        "avg_latency_ms": 250.0,
        "is_blacklisted": false
    }
}

Key fields to check:

  • High failed_inferences → peer may be unreliable
  • is_blacklisted: true → peer's reputation dropped below 0.3
  • Low avg_tokens_per_sec → peer has slow hardware

ledger.json

Stores the credit ledger:

{
    "12D3KooW...": {
        "credit_balance": 98.5,
        "total_earned_credits": 45.0,
        "total_spent_credits": 46.5,
        "total_served_tokens": 450,
        "total_queried_tokens": 465
    }
}

Key fields to check:

  • Negative credit_balance near -50 → peer may be blocked from querying
  • total_earned_credits == 0 → peer is only consuming, not contributing

Benchmarking

Run the built-in benchmark suite to baseline performance:

./target/release/daemon benchmark --iterations 100

Measures:

  • Embedding generation — ONNX model inference latency
  • SHA-256 hashing — Used for DHT keys and model verification
  • JSON serialization — Message encoding/decoding speed
  • LSH bitmask matching — Hamming distance calculation
  • Ed25519 signing — Token packet signature generation

Reports P50, P95, P99 latencies for each operation.

Common Debug Workflows

"Why isn't my query reaching remote experts?"

  1. Check peer count: curl http://127.0.0.1:8080/v1/peers | python -m json.tool
  2. If 0 peers, check bootstrap nodes in config
  3. Enable P2P debug logging: RUST_LOG=daemon::p2p=debug
  4. Look for Kademlia: GetProviders events — if no providers found, increase max_hamming_radius
  5. Check that your expert description is specific enough for LSH routing

"Why is the response quality poor?"

  1. Check which expert(s) handled the query (visible in streaming responses via expert_peer_id)
  2. Verify the expert's description matches the query domain
  3. Check the aggregation strategy used — RUST_LOG=daemon::coordinator=debug
  4. Try querying with temperature: 0.0 for deterministic output
  5. Run the query directly against your local model (bypass swarm routing) to compare

"Why is a peer getting blacklisted?"

  1. Check state/reputation.json for the peer's poi_fail count
  2. If PoI audits are failing, the peer may be:
    • Running a different model than advertised
    • Using non-deterministic settings (temperature > 0) during audits
    • Experiencing hardware errors producing inconsistent outputs
  3. Enable reputation logging: RUST_LOG=daemon::reputation=debug

"Why is inference slow?"

  1. Run benchmarks: ./target/release/daemon benchmark --iterations 50
  2. Check if the model is loaded: look for "Model loaded" in logs
  3. Verify backend setting: cpu uses all available cores; cuda offloads to GPU
  4. Check max_context — larger contexts use more memory and are slower
  5. Monitor system resources during inference (CPU, RAM, VRAM usage)

GUI Debugging

The Tauri GUI has its own debug tools:

  1. DevTools: Right-click in the app window → "Inspect Element" to open browser DevTools
  2. Console: Check the browser console for Svelte errors and IPC failures
  3. Backend logs: The sidecar daemon logs are available in the GUI's Dashboard
  4. gRPC connection: The GUI connects to the daemon via gRPC on port 50051 — check that port is available

Related: