Storage, Git Sync, and Linking Architecture

[perber]

Current State

LeafWiki is designed to be reconstructable from the files stored in a workspace.

Markdown files are the primary source of truth. LeafWiki-specific metadata such as leafwiki_id, leafwiki_pinned, and related properties is stored in the Markdown frontmatter and read when the application starts.

In addition to the Markdown files, a LeafWiki workspace currently contains:

• an assets directory for attachments and other file-based resources
• a .leafwiki directory for revisions and internal application data

Because LeafWiki relies heavily on Markdown files, bidirectional synchronization with Git repositories or external filesystem-based tools could be a valuable feature.

However, this also introduces several architectural questions and potential problems.

Git and Bidirectional Synchronization

A bidirectional Git integration would allow users to edit files both through LeafWiki and through external tools.

The main concern is merge conflicts.

Conflicts may occur when:

• a file is modified in LeafWiki and externally at the same time
• frontmatter metadata is changed independently from the Markdown content
• files are renamed or moved outside LeafWiki
• multiple files receive the same leafwiki_id
• a file is deleted on one side and modified on the other
• generated or internal metadata changes create unnecessary Git diffs
• revisions stored in .leafwiki diverge from the current file state

We need to determine how LeafWiki should detect and resolve these situations.

Some open questions are:

• Should LeafWiki rely on Git’s normal conflict markers?
• Should frontmatter and document content be merged separately?
• Should LeafWiki provide a dedicated conflict-resolution interface?
• Which metadata fields should be considered user-editable?
• Should some metadata be excluded from the Markdown files?
• How should file moves and renames be detected?
• How should LeafWiki react to invalid or duplicated IDs?
• Should synchronization be event-based, scheduled, or manually triggered?

Currently, files can be rescanned through the maintenance window. This may work as a manual recovery mechanism, but it may not be sufficient for continuous bidirectional synchronization.

Filesystem Agent Systems

Filesystem agents and other external automation tools introduce similar challenges, even without Git.

An external process may:

• create new Markdown files
• modify existing files
• rename or move files
• delete files
• update frontmatter
• create temporary or partially written files
• modify multiple related documents without using a transaction

LeafWiki must decide how it handles changes that happen while the application is running.

Important questions include:

• Should LeafWiki watch the filesystem continuously?
• How do we avoid reading files while they are still being written?
• How do we distinguish intentional changes from temporary editor files?
• How should bulk changes be processed?
• Should rescanning rebuild the entire tree or update it incrementally?
• How should unsupported or malformed files be reported?
• What happens when an external tool modifies LeafWiki-managed metadata?

The filesystem is currently the source of truth, but external agents make it necessary to define clearer ownership and consistency rules.

Linking

Another area that requires attention is the linking system.

LeafWiki currently represents links as HTTP-style addresses. This may cause problems when linking documents locally or when a workspace is moved, cloned, or opened on another machine.

For example, links may need to work across:

• the LeafWiki web application
• local Markdown editors
• Git repository viewers
• filesystem-based tools
• exported or published versions of a workspace

Possible approaches could include:

• relative Markdown links
• workspace-relative links
• stable links based on leafwiki_id
• custom LeafWiki URI schemes
• a separation between stored link representation and rendered URL
• automatic link rewriting when files are moved

We should also define what happens when:

• a target file is renamed
• a target file is moved
• two files have the same name
• a link points to a missing document
• a document is opened outside LeafWiki
• links target assets rather than Markdown documents

Main Components Affected

The main areas that would likely need to be reviewed are:

• node_store
• tree_service
• the Markdown package
• the links subsystem

The node_store, tree_service, and Markdown package currently read and reconstruct data from the filesystem. These components would need clear responsibilities around:

• filesystem change detection
• parsing and validation
• identity management
• conflict detection
• incremental updates
• error recovery

The links subsystem would need a consistent model for resolving links independently of whether a document is accessed through HTTP, Git, or the local filesystem.

Scope

MCP support is planned, but it is outside the scope of this discussion.

For now, the main questions are:

1. What problems should we expect when integrating LeafWiki with Git?
2. What problems arise when external filesystem agents modify a LeafWiki workspace?
3. How should conflicts, file identity, renames, and deletions be handled?
4. What should the canonical link format be?
5. Which parts of the current storage model should remain file-based, and which parts may require a separate index or database?

Feedback, architectural ideas, and examples from similar systems are very welcome.

[perber]

@designermonkey: Here are many questions. Not all need answer, but I guess it is a good starting point.

[perber]

I think the first topic we should address is link and asset handling, as both are fundamental to portability and future Git or filesystem synchronization.

[designermonkey]

I've been investigating a few things, including what's in this discussion so far, and have done some research in the codebase to see what should be implemented in what order. I have more detailed information on these to share at a later date, but for now here's some really high-level insight.

I'm unsure how you want to handle dependency collision, so I thought it best to figure out what affects what before making any decisions.

1. Frontmatter (leafwiki_* → leafwiki: {})

something I have been thinking about

Feasible, backwards-compatible. Dual-format parser reads both; writer emits only new. ⚠️ Security gap: IsSystemKey() doesn't block bare key leafwiki — must fix first.

2. Storage Model

Hybrid is right (FS for content, SQLite for indexes). .order.json → move to tree.db with position column. Write-through cache, FS always source of truth.

3. Link Format

HTTP absolute paths ([Page](/docs/api)) break in editors/git viewers. Store as relative paths + promote [[Title]] wikilinks. Add leafwiki://{id} for rename-proof links.

4. Conflict Handling

No merge system exists — git aborts, rescan overwrites. Need ConflictResolver with typed conflicts + configurable strategies. Start with last-write-wins, add 3-way merge later.

5. Git Sync

One-way push exists. Need pull flow: fetch → detect changes → resolve conflicts → resync. Match external files by leafwiki_id (primary) or path.

6. Filesystem Watch

No auto-detection of external edits. Add fsnotify watcher + debounce + incremental tree updates. Opt-in (--fs-watch). Pause during git pull.

7. Plugin Architecture

something I have been thinking about

Low-hanging fruit — 5 hook points already exist (PageSideEffect, RouteRegistrar, DialogRegistry, etc.). Formal interface wraps existing hooks, no refactor needed.

---

Implementation Order

internal/core/tree/node_store.go is touched by 6 of 7 plans. Order:

1. Frontmatter
2. Storage model
3. Plugins
4. Link format
5. Conflict handling
6. Git sync
7. Filesystem watch.

---

I can always remov the things not in this original discussion if you would prefer.

[perber]

I would prefer to go with the asset and link handling first. I have the feeling this is the hard ground work. The other will not be easy, but can be added on top later.

What do you mean by dependency collision?

Three-way merge and stuff is in my opinion for a other stage of the project.

[designermonkey]

Oh, I meant that some features will need rework if done before others; bad choice of wording there (it was late :D )

[designermonkey]

Here's a plan for links and assets:

Plan: Canonical Link Format — LeafWiki

Date: 2026-07-31
Context: Discussion #1315

---

Problem

HTTP-style absolute paths in markdown ([Page](/docs/api)) break when wiki is viewed in external editors, git web, or moved between workspaces. Links use paths, not stable IDs — renaming a page breaks all links to it.

Assets (images, PDFs, etc.) placed manually alongside wiki pages are linked via relative paths that work in editors but 404 in the web app — browser resolves them against the page URL, and the server doesn't serve arbitrary filesystem paths.

---

Design Constraints

Constraint	Requirement
Works in LeafWiki web app	Links must resolve in browser
Works in external editors	Links must be valid relative markdown
Works in git web (GitHub/GitLab)	Links must resolve in repo file tree
Survives workspace move/clone	Links must not break when root path changes
Survives page rename/move	Links must survive structural changes
Backward compatible	Existing links continue working
OKF-compatible	Supports absolute bundle-relative paths (/tables/customers.md)

---

Proposed Link Formats

Format 1: Relative markdown paths (primary recommendation)

Store links as tree-relative paths. Works everywhere — editors, git web, LeafWiki web.

[Page](../target-page.md)
[Section](../../other/target-page.md#section-heading)

Key principle: storage ≠ rendering.

• Write: relative path
• Read (web): resolve relative → wiki route path → render as <a href="/route/path">
• Read (editor/git): relative path works natively

Format 2: [[Title]] wiki-links (recommended for internal linking)

Already supported. Resolved by title, stable across renames. Compatible with Obsidian, Foam.

[[API Reference]]
[[API Reference|docs]]

Proposal: Promote to default. "Link to page" dialog inserts [[Title]] by default. Autocomplete on [[ typing.

Format 3: leafwiki://{id} (rename-proof stable links)

Custom URI scheme using leafwiki_id as stable identifier. Survives renames, moves, cross-workspace copy.

[Page](leafwiki://abc123)

Resolved by LinkService.ResolveLeafWikiURI() — looks up current route path by page ID. Renders as resolved <a href="/current/path"> in web app, opaque URI elsewhere.

Format 4: Absolute bundle-relative paths (OKF support)

OKF-specified format. Already supported by codebase. Not deprecated.

[Page](/tables/customers.md)

---

Resolution Order (all formats)

When rendering a link, try in order:

1. leafwiki://{id} → resolve by ID, return route path
2. [[Title]] → resolve by title, return route path
3. Relative path → resolve against current page path
4. Absolute path /docs/api → route lookup
5. File reference → resolved path is not a page → check file on disk → serve from /wiki/
6. External → pass through

---

File References (Non-Page Assets)

Images, PDFs, audio, video placed manually in the workspace (not uploaded via editor).

Format: relative markdown paths

![Diagram](../my-images/diagram.png)
[PDF](../../files/report.pdf)

Resolution: When a link resolves to a path that's not a page, check if the file exists on disk under <storageDir>/root/. If yes → serve from /wiki/{path}. If no → broken.

Not indexed in links DB. File references aren't pages — not stored, not counted as broken links, not shown in backlinks UI.

Managed assets unchanged. Editor uploads continue using /assets/<UUID>/ — stable, rename-proof, same as today.

---

Migration Phases

Phase 1: File reference serving

Make local files display in web app. No link format changes yet.

Task	Files
Add /wiki/ static route serving <storageDir>/root/	internal/wiki/assets/routes.go or new
Auth-gate same as /assets/	internal/wiki/assets/routes.go
Path traversal protection	Route handler
Update MarkdownImage to resolve relative paths via resolveWikiLinkPath()	ui/.../MarkdownImage.tsx
Update MarkdownImage to route non-asset images to /wiki/	ui/.../MarkdownImage.tsx
Update MarkdownLink to detect file refs (not pages) and route to /wiki/	ui/.../MarkdownLink.tsx
Update link pipeline to skip indexing for file refs	internal/links/helpers.go

Phase 2: Add leafwiki:// resolution

Task	Files
Add ResolveLeafWikiURI() to LinkService	internal/links/link_service.go
Add leafwiki:// detection in render path	Frontend markdown renderer
Store leafwiki:// as link format in "Copy link" dialog	internal/wiki/links/use_cases.go
Render leafwiki:// as resolved route path in HTML	ui/.../MarkdownPreview.tsx

Phase 3: Relative path storage

Convert stored links from HTTP absolute to tree-relative in markdown files.

Task	Files
Convert link storage from HTTP absolute to tree-relative	internal/links/helpers.go
Add rewrite pass for existing absolute links	internal/links/link_refactor.go
Update link extraction to preserve relative format	internal/links/helpers.go
Validate relative links in external editors	Manual test

Phase 4: Default [[Title]] linking

Task	Files
"Link to page" dialog defaults to [[Title]]	ui/.../editor/...
"Copy link" copies [[Title]] or leafwiki://	internal/wiki/links/use_cases.go
Auto-suggest [[Title]] on [[ typing	ui/.../editor/...

---

Open Decisions

1. leafwiki:// user-visible or internal? — Visible. Enables cross-workspace links. Users can copy/paste between wikis.

2. Hash fragments on leafwiki://? — Not needed initially. leafwiki://id resolves to page, browser handles #heading on the resolved URL.

3. Phase ordering? — Phase 1 (file serving) is a prerequisite for relative paths working in web app. Phases 2-4 can be reordered depending on priority.

4. wikilink: sentinel prefix? — Keep it. Disambiguates title-based links from path-based links in the DB.

[perber]

@designermonkey Thanks. I need to go over it when I have a quiet moment.

[perber]

Hi @designermonkey,

I've thought about it over the last few days, and I don't think I want to go down this route.

It would add quite a bit of complexity to the system and, at least for me, doesn't really fit the scope of a wiki. I think tools like OpenNotebook are better suited for that kind of workflow, as they are specifically designed for it.

I'd rather keep LeafWiki focused and use dedicated tools for functionality that is primarily related to working with files.

So I would keep the model where LeafWiki is the primary writer/editor. I'll adapt the storage and linking model so that files and links also work properly outside of LeafWiki.