nuwiki/SPEC.md

# nuwiki — Project Specification

> Last updated: 2026-05-10
> Status: Phase 3 (Vimwiki Lexer) complete — moving to Phase 4 (Vimwiki Parser)

---

## 1. Overview

nuwiki is a Vim/Neovim plugin that provides full vimwiki syntax support, implemented as a Rust-based Language Server (LSP). It is a spiritual successor to vimwiki, architecturally distinct from it, designed for correctness, performance, and extensibility.

**Goals:**
- Full vimwiki syntax support (highlighting, navigation, link following, diagnostics)
- First-class support for both Vim 9+ and Neovim 0.5+
- Installable via standard plugin managers (lazy.nvim, vim-plug, Dein)
- Extensible to Markdown syntax without changes to core architecture
- Independently testable core library with no editor dependencies

**Non-goals (v1.0):**
- MediaWiki syntax support
- Multi-wiki configurations
- Incremental / tree-sitter parsing
- Browser-based or WASM build

---

## 2. Repository

| Property | Value |
|---|---|
| License | Dual MIT/Apache-2.0 |
| MSRV | Rust 1.83 (stable-2) |
|---|---|
| URL | `https://code.gfran.co/gffranco/nuwiki` |
| Version control | Gitea |
| CI/CD | Gitea Actions |

---

## 3. Naming Conventions

| Context | Name |
|---|---|
| GitHub/Gitea repo | `nuwiki` |
| Cargo workspace | `nuwiki` |
| Core library crate | `nuwiki-core` |
| LSP bridge crate | `nuwiki-lsp` |
| Binary crate | `nuwiki-ls` |
| Vim plugin entry | `plugin/nuwiki.vim` |
| Lua module | `require('nuwiki')` |
| VimL autoload | `nuwiki#lsp#start()` |
| Vim help file | `doc/nuwiki.txt` → `:h nuwiki` |
| Release binary | `nuwiki-ls-{version}-{target}.tar.gz` |

---

## 4. Technology Stack

| Concern | Choice | Rationale |
|---|---|---|
| Implementation language | Rust | Performance, type safety, single binary distribution, ideal for AST modelling |
| Editor integration | LSP over stdio | Works in both Vim and Neovim without a shared scripting language |
| LSP server library | `tower-lsp` | Async, tokio-based, higher-level than `lsp-server`; easier to start with |
| Parser library | `winnow` | Successor to `nom`, designed for resilient/partial parsing; strong error recovery |
| Cargo edition | 2021 | Current standard; required for resolver v2 |
| Cargo resolver | v2 | Required for edition 2021 workspaces |
| Vim glue layer | VimL (`plugin/nuwiki.vim`, `autoload/`) | Universal entry point for all plugin managers |
| Neovim glue layer | Lua (`lua/nuwiki/`) | First-class Neovim API access via `vim.lsp.start()` |

---

## 5. Repository Layout

```
nuwiki/
│
├── Cargo.toml                  # Rust workspace root
├── Cargo.lock
│
├── crates/
│   ├── nuwiki-ls/              # Binary crate — thin main.rs, starts stdio LSP server
│   │   ├── Cargo.toml
│   │   └── src/main.rs
│   │
│   ├── nuwiki-core/            # Library crate — parser, AST, renderer (no editor deps)
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs
│   │       ├── syntax/
│   │       │   ├── mod.rs
│   │       │   ├── registry.rs
│   │       │   └── vimwiki/
│   │       │       ├── mod.rs
│   │       │       ├── lexer.rs
│   │       │       └── parser.rs
│   │       ├── ast/
│   │       │   ├── mod.rs
│   │       │   ├── block.rs
│   │       │   ├── inline.rs
│   │       │   └── link.rs
│   │       └── render/
│   │           ├── mod.rs
│   │           └── html.rs
│   │
│   └── nuwiki-lsp/             # Library crate — LSP protocol bridge
│       ├── Cargo.toml
│       └── src/lib.rs
│
├── plugin/                     # Universal Vim/Neovim entry point
│   └── nuwiki.vim
│
├── lua/                        # Neovim-specific Lua layer
│   └── nuwiki/
│       ├── init.lua
│       ├── config.lua
│       ├── lsp.lua
│       └── install.lua         # Binary download / build-from-source logic
│
├── autoload/                   # Lazy-loaded VimL (Vim compat layer)
│   └── nuwiki/
│       └── lsp.vim
│
├── ftdetect/                   # Filetype detection for .wiki files
│   └── nuwiki.vim
│
├── ftplugin/                   # Per-filetype buffer settings
│   └── nuwiki.vim
│
├── syntax/                     # Static fallback syntax highlighting (no LSP required)
│   └── nuwiki.vim
│
├── doc/                        # Vim help documentation
│   └── nuwiki.txt
│
├── scripts/
│   └── download_bin.vim        # VimL binary download (used by Dein/vim-plug build hooks)
│
└── .gitea/
    └── workflows/
        ├── ci.yaml             # Lint, test, fmt check on every push/PR
        └── release.yaml        # Cross-compile + release on v* tag
```

---

## 6. Architecture

### 6.1 Layer Model

```
┌─────────────────────────────────────────────────────┐
│                   Consumer Layer                    │
│        (editor highlight, HTML export, TOC, …)      │
└────────────────────────┬────────────────────────────┘
                         │  operates on
                         ▼
┌─────────────────────────────────────────────────────┐
│              AST  (nuwiki-core, shared)              │
│        Document > Block nodes > Inline nodes        │
└────────────────────────┬────────────────────────────┘
                         │  produced by
                         ▼
┌─────────────────────────────────────────────────────┐
│         Parser — syntax-specific (nuwiki-core)      │
│      Registered per syntax; receives TokenStream    │
└────────────────────────┬────────────────────────────┘
                         │  consumes
                         ▼
┌─────────────────────────────────────────────────────┐
│         Lexer — syntax-specific (nuwiki-core)       │
│         Registered per syntax; receives raw text    │
└────────────────────────┬────────────────────────────┘
                         │  reads
                         ▼
                    Raw text input
```

### 6.2 Crate Dependency Rules

```
nuwiki-ls  →  nuwiki-lsp  →  nuwiki-core
```

- `nuwiki-core` must never depend on `nuwiki-lsp` or `nuwiki-ls`
- `nuwiki-lsp` must never depend on `nuwiki-ls`
- The VimL and Lua editor layers must never contain logic — pure wiring only

### 6.3 Syntax Plugin Interface

Every syntax (vimwiki, future markdown) implements:

```
SyntaxPlugin
  id:             string          -- "vimwiki" | "markdown"
  display_name:   string
  file_extensions: string[]       -- [".wiki"] | [".md"]
  lexer:          fn(text) → TokenStream
  parser:         fn(TokenStream) → DocumentNode
```

Registry:
```
SyntaxRegistry
  register(plugin: SyntaxPlugin)
  get(id: &str) → Option<&SyntaxPlugin>
  detect_from_extension(ext: &str) → Option<&SyntaxPlugin>
```

All plugins must be `Send + Sync` (held in the LSP server registry).

### 6.4 AST Node Types

#### Document
```
DocumentNode
  children:  Vec<BlockNode>
  metadata:  PageMetadata        -- title, nohtml, template, date
```

#### Block Nodes
```
HeadingNode        level: 1–6 | children: Vec<InlineNode> | centered: bool
ParagraphNode      children: Vec<InlineNode>
HorizontalRuleNode
BlockquoteNode     children: Vec<BlockNode>
PreformattedNode   content: String | language: Option<String>
MathBlockNode      content: String | environment: Option<String>
ListNode           ordered: bool | symbol: ListSymbol | items: Vec<ListItemNode>
DefinitionListNode items: Vec<DefinitionItemNode>
TableNode          rows: Vec<TableRowNode> | has_header: bool
CommentNode        content: String
ErrorNode          raw: String | message: String   -- resilient parse failure
```

#### Inline Nodes
```
TextNode           content: String
BoldNode           children: Vec<InlineNode>
ItalicNode         children: Vec<InlineNode>
BoldItalicNode     children: Vec<InlineNode>
StrikethroughNode  children: Vec<InlineNode>
CodeNode           content: String
SuperscriptNode    children: Vec<InlineNode>
SubscriptNode      children: Vec<InlineNode>
MathInlineNode     content: String
KeywordNode        keyword: Keyword  -- TODO|DONE|STARTED|FIXME|FIXED|XXX
ColorNode          color: String | children: Vec<InlineNode>
WikiLinkNode       target: LinkTarget | description: Option<Vec<InlineNode>>
ExternalLinkNode   url: String | description: Option<Vec<InlineNode>>
TransclusionNode   url: String | alt: Option<String> | attrs: HashMap<String,String>
RawUrlNode         url: String
```

#### Supporting Types
```
ListItemNode
  symbol:    ListSymbol
  level:     usize
  checkbox:  Option<CheckboxState>
  children:  Vec<InlineNode>
  sublist:   Option<ListNode>

DefinitionItemNode
  term:        Option<Vec<InlineNode>>
  definitions: Vec<Vec<InlineNode>>

TableRowNode   cells: Vec<TableCellNode> | is_header: bool
TableCellNode  children: Vec<InlineNode> | col_span: bool | row_span: bool

LinkTarget
  kind:         LinkKind   -- Wiki|Interwiki|Diary|File|Local|Raw|AnchorOnly
  path:         Option<String>
  wiki_index:   Option<usize>
  wiki_name:    Option<String>
  anchor:       Option<String>
  is_absolute:  bool
  is_directory: bool

ListSymbol    = Dash | Star | Hash | Numeric | NumericParen
              | AlphaParen | AlphaUpperParen | RomanParen | RomanUpperParen
CheckboxState = Empty | Quarter | Half | ThreeQuarters | Done | Rejected
Keyword       = Todo | Done | Started | Fixme | Fixed | Xxx
```

### 6.5 Span / Source Location

Every AST node carries a `Span`:

```rust
struct Span {
    start: Position,
    end:   Position,
}

struct Position {
    line:   u32,   -- 0-indexed
    column: u32,   -- 0-indexed, byte offset within line
    offset: usize, -- absolute byte offset from document start
}
```

Spans are required for LSP diagnostics, semantic tokens, and go-to-definition.

### 6.6 Lexer Strategy

- **Two-pass:** block-level pass first, then inline pass within each block
- **Token types:** syntax-specific (`VimwikiToken`) — not shared across syntaxes
- **TokenStream:** eager `Vec<Token>` wrapped in a `TokenStream` newtype (changeable to lazy later)
- Operates on Rust `char`s for correctness; spans stored as byte offsets

### 6.7 Parser Strategy

- Uses `winnow` parser combinator library
- Resilient: on malformed input, emits `ErrorNode` and continues — never fails the whole document
- Inline marker precedence rules documented explicitly in code and tests

### 6.8 Renderer

- `Renderer` trait: writer-based (`fn render(&self, doc: &DocumentNode, w: &mut dyn Write)`)
- `HtmlRenderer` is the first implementation
- Link resolution injected as a callback at construction time
- Template support: minimal `{{content}}` / `{{title}}` token substitution
- CSS: linked external stylesheet (matches vimwiki convention)

### 6.9 LSP Features

| Feature | LSP Method |
|---|---|
| Syntax highlighting | `textDocument/semanticTokens/full` + `/range` |
| Diagnostics (broken links, parse errors) | `textDocument/publishDiagnostics` |
| Follow link / go to definition | `textDocument/definition` |
| Backlinks | `textDocument/references` |
| TOC / outline | `textDocument/documentSymbol` |
| Link preview | `textDocument/hover` |
| Link autocomplete | `textDocument/completion` (trigger: `[[`) |
| Page rename | `workspace/rename` |
| Workspace search | `workspace/symbol` |

### 6.10 Document Store

```
DocumentStore: Arc<DashMap<Url, DocumentState>>

DocumentState
  text:     String
  ast:      DocumentNode
  version:  i32
```

Full re-parse on every `didOpen` / `didChange`. Incremental parsing deferred post-v1.

### 6.11 UTF-16 / Position Encoding

LSP positions negotiated as UTF-8 (`positionEncoding = "utf-8"`) during `initialize` where the client supports LSP 3.17+. UTF-16 conversion fallback implemented in the LSP layer for older clients. `nuwiki-core` is always UTF-8 / byte-offset internally.

---

## 7. Editor Integration

### 7.1 Plugin Manager Installation

**lazy.nvim:**
```lua
{
  "gffranco/nuwiki",
  build = "lua require('nuwiki').install()",
  ft = { "vimwiki" },
  opts = {},
}
```

**vim-plug:**
```vim
Plug 'gffranco/nuwiki', { 'do': 'vim -e -s -c "source scripts/download_bin.vim" -c "q"' }
```

**Dein:**
```vim
call dein#add('gffranco/nuwiki', {
  \ 'build': 'vim -e -s -c "source scripts/download_bin.vim" -c "q"'
  \ })
```

### 7.2 Binary Distribution

- Pre-built binaries published as Gitea release assets
- Named: `nuwiki-ls-{version}-{target}.tar.gz`
- Downloaded at install time by `lua/nuwiki/install.lua` or `scripts/download_bin.vim`
- Fallback: `cargo build --release` if download fails or `g:nuwiki_build_from_source = 1`
- Binary installed to `{plugin_dir}/bin/nuwiki-ls[.exe]`

### 7.3 Editor Detection

```vim
" plugin/nuwiki.vim
if has('nvim')
  lua require('nuwiki').setup()
else
  call nuwiki#lsp#start()
endif
```

### 7.4 Vim LSP Client Preference Order (Vim only)

1. `vim-lsp` (matoto/vim-lsp)
2. `coc.nvim`
3. Error message if neither found

### 7.5 User Config Schema

```lua
require('nuwiki').setup({
  wiki_root      = "~/vimwiki",   -- root directory of the wiki
  file_extension = ".wiki",       -- file extension to associate
  syntax         = "vimwiki",     -- "vimwiki" | "markdown" (future)
  log_level      = "warn",        -- "error"|"warn"|"info"|"debug"
})
```

### 7.6 Health Check

`:checkhealth nuwiki` verifies:
- Binary exists at `{plugin_dir}/bin/nuwiki-ls`
- Binary is executable and responds to `--version`
- LSP client is running
- Filetype detection works for `.wiki` files

---

## 8. CI/CD

### 8.1 Runner Setup

| Property | Value |
|---|---|
| System | Gitea Actions (`act_runner`) |
| Mode | Docker (jobs run in containers) |
| Docker access | Host socket mounted (`/var/run/docker.sock`) |
| Internet access | Yes |

### 8.2 Workflow Files

**`.gitea/workflows/ci.yaml`** — triggers on every push and PR:
- `cargo fmt --check`
- `cargo clippy -- -D warnings`
- `cargo test --workspace`

**`.gitea/workflows/release.yaml`** — triggers on `v*` tag push:
- Cross-compile for all 4 Linux targets using `cross`
- Package binaries as `.tar.gz`
- Create Gitea release and upload assets via REST API using `RELEASE_TOKEN` secret
- Publish `nuwiki-core` to crates.io using `CARGO_REGISTRY_TOKEN` secret

### 8.3 Build Targets

| Target | Built by |
|---|---|
| `x86_64-unknown-linux-gnu` | CI (`cross`) |
| `aarch64-unknown-linux-gnu` | CI (`cross`) |
| `x86_64-unknown-linux-musl` | CI (`cross`) |
| `aarch64-unknown-linux-musl` | CI (`cross`) |
| `x86_64-apple-darwin` | Manual |
| `aarch64-apple-darwin` | Manual |
| `x86_64-pc-windows-msvc` | Manual |

### 8.4 Release Secrets

| Secret | Purpose |
|---|---|
| `RELEASE_TOKEN` | Gitea personal access token for creating releases and uploading assets |
| `CARGO_REGISTRY_TOKEN` | crates.io API token for publishing `nuwiki-core` |

### 8.5 `actions/cache` Configuration

Requires `act_runner`'s `config.yaml` to have the cache server host set to the runner host's LAN IP so job containers can reach it.

### 8.6 Breaking Change Policy

A semver-breaking change (`v0.x` → `v0.x+1` or `v1.x` → `v2.0`) is defined as any of:
- AST node type additions, removals, or field changes in `nuwiki-core`
- `SyntaxPlugin` or `Renderer` trait signature changes
- User config schema key removals or type changes
- Removed LSP capabilities

---

## 9. Vimwiki Syntax Feature Checklist

### Typefaces
- [ ] Bold: `*text*`
- [ ] Italic: `_text_`
- [ ] Bold italic: `_*text*_` / `*_text_*`
- [ ] Strikethrough: `~~text~~`
- [ ] Inline code: `` `text` ``
- [ ] Superscript: `super^script^`
- [ ] Subscript: `sub,,script,,`
- [ ] Keywords: `TODO` `DONE` `STARTED` `FIXME` `FIXED` `XXX`

### Links
- [ ] Plain wikilink: `[[Target]]`
- [ ] Described wikilink: `[[Target|Description]]`
- [ ] Subdirectory wikilink: `[[dir/Page]]`
- [ ] Root-relative wikilink: `[[/Page]]`
- [ ] Filesystem-absolute wikilink: `[[//path]]`
- [ ] Subdirectory link: `[[dir/]]`
- [ ] Interwiki numbered: `[[wiki1:Page]]`
- [ ] Interwiki named: `[[wn.Name:Page]]`
- [ ] Diary link: `[[diary:YYYY-MM-DD]]`
- [ ] Anchor-only link: `[[#Anchor]]`
- [ ] Wikilink with anchor: `[[Page#Anchor]]`
- [ ] Raw URLs: `https://…` `mailto:…` `ftp://…`
- [ ] External file link: `[[file:path]]` / `[[local:path]]`
- [ ] Transclusion: `{{URL}}` with optional alt and attrs
- [ ] Thumbnail link: `[[imgURL|{{thumbURL}}]]`

### Headers
- [ ] Levels 1–6: `= H1 =` … `====== H6 ======`
- [ ] Centered header (leading whitespace before `=`)

### Lists
- [ ] Unordered: `-` and `*`
- [ ] Ordered: `1.` `1)` `a)` `A)` `i)` `I)` `#`
- [ ] Nested and mixed types
- [ ] Multi-line items (indentation continuation)
- [ ] Definition lists: `Term:: Definition`
- [ ] Checkboxes: `[ ]` `[.]` `[o]` `[O]` `[X]` `[-]`

### Tables
- [ ] Basic `|`-delimited table
- [ ] Header row separator: `|---|`
- [ ] Column span: `>`
- [ ] Row span: `\/`
- [ ] Inline formatting inside cells

### Preformatted Text
- [ ] Fenced block: `{{{ … }}}`
- [ ] Optional language/class on opening fence

### Mathematical Formulae
- [ ] Inline math: `$ … $`
- [ ] Block display math: `{{$ … }}$`
- [ ] Block environment math: `{{$%env% … }}$`

### Blockquotes
- [ ] 4-space indent blockquote
- [ ] `>` prefix blockquote

### Comments
- [ ] Single-line: `%% …`
- [ ] Multi-line: `%%+ … +%%`

### Horizontal Rule
- [ ] Four or more dashes: `----`

### Placeholders
- [ ] `%title <text>`
- [ ] `%nohtml`
- [ ] `%template <name>`
- [ ] `%date <date>`

---

## 10. Implementation Phases

| Phase | Name | Key Output |
|---|---|---|
| 0 | Scaffolding | Compiling empty workspace + CI skeleton |
| 1 | Core AST | All node types, spans, Visitor trait |
| 2 | Syntax Plugin Interface | `Lexer`/`Parser`/`SyntaxPlugin` traits, `SyntaxRegistry` |
| 3 | Vimwiki Lexer | `VimwikiToken`, two-pass lexer with spans |
| 4 | Vimwiki Parser | Full AST from token stream, error recovery |
| 5 | Renderer | `Renderer` trait + `HtmlRenderer` |
| 6 | LSP Foundation | `didOpen`/`didChange`, diagnostics, document symbols |
| 7 | Semantic Tokens | AST → LSP semantic token stream |
| 8 | Navigation | Definition, references, hover, completion, workspace index |
| 9 | Editor Glue | Installable plugin, health check, binary download |
| 10 | CI/CD | Automated lint/test/release pipeline |

---

## 11. Pending Decisions

These decisions are required before or during the phase indicated.

| # | Decision | Needed by | Options | Notes |
|---|---|---|---|---|
| ~~P1~~ | ~~**License**~~ | ~~Phase 0~~ | ✅ **Dual MIT/Apache-2.0** | |
| ~~P2~~ | ~~**MSRV**~~ | ~~Phase 0~~ | ✅ **stable-2 (Rust 1.83)** | |
| ~~P3~~ | ~~**String representation in AST**~~ | ~~Phase 1~~ | ✅ **`String` (owned)** | |
| ~~P4~~ | ~~**Visitor pattern**~~ | ~~Phase 1~~ | ✅ **Defined in Phase 1** | Open-recursion `Visitor` trait + `walk_*` helpers |
| P5 | **Minimum Neovim version** | Phase 9 | 0.8 (`vim.lsp.start`) · 0.11 (`vim.lsp.config`) | 0.8 = broader compat; 0.11 = cleaner Lua API |
| P6 | **Minimum Vim version** | Phase 9 | Vim 8.2 · Vim 9.0 · Vim 9.1 | LSP client plugins work best on 9.0+; recommend documenting 9.1 as minimum |
| P7 | **Semantic token type mapping** | Phase 7 | Standard LSP types only · Custom token types | Custom types need client-side highlight group definitions; more flexible but more setup for users |
| P8 | **Workspace indexing strategy** | Phase 8 | Eager on startup · Lazy + background | Eager blocks startup on large wikis; recommend lazy with progress notification |
| P9 | **macOS runner** | Phase 10 | Stay manual · Register macOS runner
| Revisit if a Mac is available; no action needed now |