Skip to content

repo_guide.md

Latest commit

 

History

History
186 lines (148 loc) · 11.6 KB

repo_guide.md

File metadata and controls

186 lines (148 loc) · 11.6 KB
Raw

DiffMem Repository Guide

This guide defines the structure, creation, and maintenance of a DiffMem memory repository. The core is plain-text Markdown files in a Git repo, leveraging Git's version control for temporal insights. Designed as an external "hippocampus" for memory augmentation, it simulates brain-like processes: hierarchical organization (lifetime themes, general events, specific episodes), reconstructive retrieval (pattern completion from cues), event boundaries for temporal cohesion, semantic spreading via links, and adaptive strengthening (plasticity via edit frequency).

Key benefits vs semantic/vector/graph models:

  • Editable Current-State: Files maintain a concise "now" view, editable to reflect evolution without sprawl, ensuring quick loads for 50-year durability and users with memory challenges (fits LLM token limits ~50k).
  • Blame Optimization: Traces origins of current lines/blocks (e.g., "When was this fact last updated?"). For full historical snapshots or evolution, use git log -p, git diff, or git show to reconstruct past states.
  • Context Balance: Bounded blocks (headers to /END) for targeted queries; files under 2000 lines to handle long arcs.
  • Memory Differentiation (Inspired by SMS Model):
    • Biographical Memory (Lifetime Periods): Broad, thematic self-identity (years-scale).
    • Episodic Memory (General Events & Event-Specific Knowledge): Chronological sequences (months/weeks/days) and unique details (hours/minutes).
    • Factual/Semantic Memory: Timeless concepts/facts, with spreading activation via links.
  • Reconstructive Elements: Supports direct/generative retrieval; pattern completion via git tools.
  • Git Harmony: Buffered, serialized writes prevent conflicts. Monthly merges use diffs for summaries; periodic pruning condenses stale data.
  • Permissioning: Each user/agent has an isolated folder; access controlled by folder permissions (e.g., OS-level or Git remote auth).

Follow this guide to create and update files. All changes are committed via DiffMem's buffered workflow: edits accumulate in the working tree during a session, then a single commit seals them with a descriptive message.

Architecture: Orphan Branches & Worktrees

DiffMem uses a Federated Git Architecture to ensure strict isolation between users while maintaining a scalable central storage.

  1. Storage Repository: A single, hidden Git repository (the "Library Stack") holds all history for all users.
    • User data lives on isolated Orphan Branches (e.g., user/alex, user/anna).
    • These branches share no history or files.
  2. Active Contexts (Worktrees): When a user is active, their branch is checked out into a Git Worktree (the "Reading Room").
    • The agent operates inside this worktree.
    • To the agent, this folder looks like the root of a standard Git repo.
    • Path: /app/active_contexts/{user_id}/ (or configured root).

Repository Structure (Per User Context)

When a user context is mounted, the directory structure is flat and simple. The users/ nesting is gone.

<worktree_root>/           # Root of the user's isolated environment
├── repo_guide.md          # This file
├── index.md               # Autogenerated: quick lookup hints
├── memories/              # Core memory storage
│   ├── people/            # Biographical: profiles
│   ├── contexts/          # Factual/semantic: themes
│   └── episodes_index.md  # Autogenerated: thematized groupings
├── timeline/              # Episodic: chronological
│   └── YYYY-MM.md         # Monthly timeline files
└── .git                   # File pointing to central storage (Git internal)
  • memories/ Subfolder: Holds all memory files. Keep flat to minimize depth.
  • index.md: Per-user/agent, autogenerated. Keyword summaries and links (e.g., "anxiety → contexts/health.md#milestones").
  • episodes_index.md: Per-user/agent, autogenerated. Thematized groupings (e.g., "Stress Cycles: Links to timeline/2024-10.md#eventX + contexts/health.md#challenges") for spreading activation and hierarchical recall.

Creating and Structuring Contexts

  1. Initialize: Use the DiffMem Server or API to onboard a user.
    • This automatically creates the orphan branch user/{id} and mounts the worktree.
  2. Folder Generation: The system auto-generates memories/, timeline/, and index.md.
  3. Avoid Sprawl: Limit subfolders to the defined ones. New themes go under contexts/ as files. Target: 5-20 files per user for 50-year scalability.

File Types and Creation

Files are Markdown (.md) for readability. Create manually or via DiffMem. Use editable blocks (e.g., ### Header to /END) for current-state focus—update in-place to reflect changes. Embed event boundaries, temporal signatures (e.g., "Drift: Post-crisis"), and strength indicators (e.g., "[Strength: High]" for frequently edited items, simulating plasticity). Differentiate by type:

1. Biographical Memory Files (Lifetime Periods)

  • Location: memories/people/<person>.md (e.g., alex.md for self). One file per key person.
  • Purpose: Evolving self-identity; editable for "now" with light evolution embeds. Supports recollection (rich details via links).
  • Creation: Start with template when mentioned.
  • Template: 
    # [Name] Profile (Biographical Core) [Strength: Medium]

## Core Identity [ALWAYS_LOAD]
- Essential markers: [e.g., "Born 1980, software engineer, introverted"]
- Key traits: [e.g., "Resilient to stress, but prone to overthinking"]

### ❤️ Relationship Dynamics [Strength: High]
• With Annabelle: Friends → deep trust → intellectual/emotional safety
  ↳ Connection Quality: Intellectual stimulation + emotional understanding + shared growth
  ↳ Interaction Pattern: Conversational, exploratory, mutually supportive
/END ❤️ Relationship Dynamics

[Additional editable blocks, e.g., ### Personality Traits]
  • Guidelines: Edit in-place; embed "↳ Evolution" for reconstructive cues. Mark [Strength: Low/Medium/High] based on edit frequency (plasticity simulation).

2. Episodic Memory Files (General Events & Event-Specific Knowledge)

  • Location: timeline/<YYYY-MM>.md (e.g., 2024-10.md). One per month.
  • Purpose: Chronological events; edit for accuracy, with boundaries for intra-event cohesion.
  • Creation: DiffMem auto-creates monthly.
  • Template: 
    # Timeline: YYYY-MM (Episodic Records) [Temporal Drift: Busy transition period]

## Monthly Summary [MERGE_LOAD]
- Overview: [e.g., "Busy month with travel and health checkups"]

### Daily Entries
#### YYYY-MM-DD (Session: [session_id]) [Strength: Medium]
- Event: [e.g., "Visited India; felt homesick"]
- Context: [e.g., "With family; emotional high"]
## Event Boundary: End of travel phase
- Links: [e.g., See people/alex.md#relationship-dynamics for trait update]
/END YYYY-MM-DD
  • Guidelines: Edit entries; add boundaries (## Event Boundary) to improve order recall within events. Include drift notes for unique signatures.

3. Factual/Semantic Memory Files

  • Location: memories/contexts/<theme>.md (e.g., health.md). One per theme.
  • Purpose: Timeless facts; editable with associative links for spreading activation.
  • Creation: Add when theme emerges (limit 10-15).
  • Template: 
    # [Theme] Facts (Factual/Semantic Reference) [Strength: Low]

## Core Facts [ALWAYS_LOAD]
- Key details: [e.g., "Chronic condition: ADHD diagnosed 2010"]

### Health Milestones
• Diagnosis: ADHD (2010)
  ↳ Evolution: Managed with therapy; recent medication change
  ↳ Current Impact: Improved focus but ongoing executive function challenges
/END Health Milestones
  • Guidelines: Edit in-place; use for semantic control (e.g., inhibit irrelevants via query orchestration).

Linking Between Files

DiffMem uses Obsidian-style wikilinks as the canonical link format. The consolidator agent (run_link) generates these automatically by mining commit co-occurrence; you generally do not need to author them by hand.

  • Format: [[memories/people/maya|Maya]] — full vault path (without .md suffix) + pipe + display name.
  • Examples:
    • [[memories/people/sam_rivera|Sam Rivera]]
    • [[memories/contexts/project_x|Project X]]
    • [[memories/people/alex|Alex]]
  • Purpose: Renders as a clickable link in Obsidian (open the worktree as a vault) AND is greppable with \[\[ for agents. Enables graph-style traversal across the memory folder.
  • Guidelines: One wikilink per target entity per file. Place inline in prose where the entity is naturally mentioned. The ## SEMANTIC INDEX section is off-limits to link insertion.
  • Legacy format (pre-consolidator): [Label](relative/path/to/file.md#anchor; assoc=type). Existing legacy links continue to work but new links should use the wikilink syntax above.

Consolidation

The consolidator agent runs out-of-band (separate from the writer's session hot path) and produces commits whose messages always start with consolidate(...). Three tools:

  • consolidate(dedupe): {survivor} ← {loser} — two entity files were merged. The loser's filename stem is preserved as an alias in the survivor's SEMANTIC INDEX so future writer-agent runs recognise it.
  • consolidate(redistribute): slim {entity} (N moves, M new contexts) — an oversized entity had sections moved to subject entities or extracted as new memories/contexts/{slug}.md files.
  • consolidate(link): wikilinks across N files (window=K) — Obsidian-style wikilinks were woven into prose based on co-occurrence in the last K commits.

Retrieval-aware agents can filter consolidation commits from session-formation commits with git log --grep='^consolidate' (or the inverse). The writer agent never produces commits with this prefix.

History Creation and Session Referencing

  • How History Builds: Buffer changes during sessions; commit atomically (msg: "Session [session_id]: Updated dynamics").
  • Session Referencing: Include (Session: [session_id])—UUID to external chat logs for verbatim recall.
  • Tracing History: Blame for current provenance (git blame -L start,end file.md). For reconstruction: git show <commit>:file.md (past snapshot) or git log -p -L start,end:file.md (evolution arc).
  • Plasticity Simulation: Frequent commits boost [Strength] tags; use for BM25 weighting (stronger memories prioritized).

Git Practices for Blame and Merges

  • Blame Usage: For current blocks (header to /END); shows last commit per line. Use log/diff for deleted content or full arcs.
  • Merge Workflow: Can support Pull Requests by pushing user/alex to GitHub and opening a PR.
  • Pruning for Longevity: Yearly "prune commits" condense stale blocks (embed summaries, archive to branches). Mimics synaptic pruning.
  • Diff Insights: git diff for changes; supports pattern completion (reconstruct from partial cues).

Balancing Context Length and Avoiding Sprawl

  • Sweet Spot: Editable blocks keep files lean; load 2-5 per query. Use [ALWAYS_LOAD] for essentials, [RECENT_LOAD] for timelines.
  • Anti-Sprawl Tips: Edit in-place; split overgrown blocks during merges/prunes. Archive old timelines in Git history.
  • Context Guidance: Mark with [ALWAYS_LOAD], [RECENT_LOAD], [KEYWORD_LOAD]. Support direct (quick BM25) vs. generative (iterative LLM) retrieval.

This structure creates a durable, brain-inspired memory system that evolves gracefully over 50 years. Update via DiffMem to preserve integrity.