nuwiki/SPEC.md

nuwiki — Project Specification

Last updated: 2026-05-10 Status: Phase 6 (LSP Foundation) complete — moving to Phase 7 (Semantic Tokens)

---

1. Overview

nuwiki is a Vim/Neovim plugin that provides full vimwiki syntax support, implemented as a Rust-based Language Server (LSP). It is a spiritual successor to vimwiki, architecturally distinct from it, designed for correctness, performance, and extensibility.

Goals:

• Full vimwiki syntax support (highlighting, navigation, link following, diagnostics)
• First-class support for both Vim 9+ and Neovim 0.5+
• Installable via standard plugin managers (lazy.nvim, vim-plug, Dein)
• Extensible to Markdown syntax without changes to core architecture
• Independently testable core library with no editor dependencies

Non-goals (v1.0):

• MediaWiki syntax support
• Multi-wiki configurations
• Incremental / tree-sitter parsing
• Browser-based or WASM build

---

2. Repository

Property	Value
License	Dual MIT/Apache-2.0
MSRV	Rust 1.83 (stable-2)
---	---
URL	https://code.gfran.co/gffranco/nuwiki
Version control	Gitea
CI/CD	Gitea Actions

---

3. Naming Conventions

Context	Name
GitHub/Gitea repo	nuwiki
Cargo workspace	nuwiki
Core library crate	nuwiki-core
LSP bridge crate	nuwiki-lsp
Binary crate	nuwiki-ls
Vim plugin entry	plugin/nuwiki.vim
Lua module	require('nuwiki')
VimL autoload	nuwiki#lsp#start()
Vim help file	doc/nuwiki.txt → :h nuwiki
Release binary	nuwiki-ls-{version}-{target}.tar.gz

---

4. Technology Stack

Concern	Choice	Rationale
Implementation language	Rust	Performance, type safety, single binary distribution, ideal for AST modelling
Editor integration	LSP over stdio	Works in both Vim and Neovim without a shared scripting language
LSP server library	tower-lsp	Async, tokio-based, higher-level than lsp-server; easier to start with
Parser library	None — hand-rolled recursive descent	Tokens are span-bearing structs that fit awkwardly into winnow/nom combinator style; resilience and error recovery are explicit in the parser code. Originally specified as winnow; revisited in Phase 4.
Cargo edition	2021	Current standard; required for resolver v2
Cargo resolver	v2	Required for edition 2021 workspaces
Vim glue layer	VimL (plugin/nuwiki.vim, autoload/)	Universal entry point for all plugin managers
Neovim glue layer	Lua (lua/nuwiki/)	First-class Neovim API access via vim.lsp.start()

---

5. Repository Layout

nuwiki/
│
├── Cargo.toml                  # Rust workspace root
├── Cargo.lock
│
├── crates/
│   ├── nuwiki-ls/              # Binary crate — thin main.rs, starts stdio LSP server
│   │   ├── Cargo.toml
│   │   └── src/main.rs
│   │
│   ├── nuwiki-core/            # Library crate — parser, AST, renderer (no editor deps)
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs
│   │       ├── syntax/
│   │       │   ├── mod.rs
│   │       │   ├── registry.rs
│   │       │   └── vimwiki/
│   │       │       ├── mod.rs
│   │       │       ├── lexer.rs
│   │       │       └── parser.rs
│   │       ├── ast/
│   │       │   ├── mod.rs
│   │       │   ├── block.rs
│   │       │   ├── inline.rs
│   │       │   └── link.rs
│   │       └── render/
│   │           ├── mod.rs
│   │           └── html.rs
│   │
│   └── nuwiki-lsp/             # Library crate — LSP protocol bridge
│       ├── Cargo.toml
│       └── src/lib.rs
│
├── plugin/                     # Universal Vim/Neovim entry point
│   └── nuwiki.vim
│
├── lua/                        # Neovim-specific Lua layer
│   └── nuwiki/
│       ├── init.lua
│       ├── config.lua
│       ├── lsp.lua
│       └── install.lua         # Binary download / build-from-source logic
│
├── autoload/                   # Lazy-loaded VimL (Vim compat layer)
│   └── nuwiki/
│       └── lsp.vim
│
├── ftdetect/                   # Filetype detection for .wiki files
│   └── nuwiki.vim
│
├── ftplugin/                   # Per-filetype buffer settings
│   └── nuwiki.vim
│
├── syntax/                     # Static fallback syntax highlighting (no LSP required)
│   └── nuwiki.vim
│
├── doc/                        # Vim help documentation
│   └── nuwiki.txt
│
├── scripts/
│   └── download_bin.vim        # VimL binary download (used by Dein/vim-plug build hooks)
│
└── .gitea/
    └── workflows/
        ├── ci.yaml             # Lint, test, fmt check on every push/PR
        └── release.yaml        # Cross-compile + release on v* tag

---

6. Architecture

6.1 Layer Model

┌─────────────────────────────────────────────────────┐
│                   Consumer Layer                    │
│        (editor highlight, HTML export, TOC, …)      │
└────────────────────────┬────────────────────────────┘
                         │  operates on
                         ▼
┌─────────────────────────────────────────────────────┐
│              AST  (nuwiki-core, shared)              │
│        Document > Block nodes > Inline nodes        │
└────────────────────────┬────────────────────────────┘
                         │  produced by
                         ▼
┌─────────────────────────────────────────────────────┐
│         Parser — syntax-specific (nuwiki-core)      │
│      Registered per syntax; receives TokenStream    │
└────────────────────────┬────────────────────────────┘
                         │  consumes
                         ▼
┌─────────────────────────────────────────────────────┐
│         Lexer — syntax-specific (nuwiki-core)       │
│         Registered per syntax; receives raw text    │
└────────────────────────┬────────────────────────────┘
                         │  reads
                         ▼
                    Raw text input

6.2 Crate Dependency Rules

nuwiki-ls  →  nuwiki-lsp  →  nuwiki-core

• nuwiki-core must never depend on nuwiki-lsp or nuwiki-ls
• nuwiki-lsp must never depend on nuwiki-ls
• The VimL and Lua editor layers must never contain logic — pure wiring only

6.3 Syntax Plugin Interface

Every syntax (vimwiki, future markdown) implements:

SyntaxPlugin
  id:             string          -- "vimwiki" | "markdown"
  display_name:   string
  file_extensions: string[]       -- [".wiki"] | [".md"]
  lexer:          fn(text) → TokenStream
  parser:         fn(TokenStream) → DocumentNode

Registry:

SyntaxRegistry
  register(plugin: SyntaxPlugin)
  get(id: &str) → Option<&SyntaxPlugin>
  detect_from_extension(ext: &str) → Option<&SyntaxPlugin>

All plugins must be Send + Sync (held in the LSP server registry).

6.4 AST Node Types

Document

DocumentNode
  children:  Vec<BlockNode>
  metadata:  PageMetadata        -- title, nohtml, template, date

Block Nodes

HeadingNode        level: 1–6 | children: Vec<InlineNode> | centered: bool
ParagraphNode      children: Vec<InlineNode>
HorizontalRuleNode
BlockquoteNode     children: Vec<BlockNode>
PreformattedNode   content: String | language: Option<String>
MathBlockNode      content: String | environment: Option<String>
ListNode           ordered: bool | symbol: ListSymbol | items: Vec<ListItemNode>
DefinitionListNode items: Vec<DefinitionItemNode>
TableNode          rows: Vec<TableRowNode> | has_header: bool
CommentNode        content: String
ErrorNode          raw: String | message: String   -- resilient parse failure

Inline Nodes

TextNode           content: String
BoldNode           children: Vec<InlineNode>
ItalicNode         children: Vec<InlineNode>
BoldItalicNode     children: Vec<InlineNode>
StrikethroughNode  children: Vec<InlineNode>
CodeNode           content: String
SuperscriptNode    children: Vec<InlineNode>
SubscriptNode      children: Vec<InlineNode>
MathInlineNode     content: String
KeywordNode        keyword: Keyword  -- TODO|DONE|STARTED|FIXME|FIXED|XXX
ColorNode          color: String | children: Vec<InlineNode>
WikiLinkNode       target: LinkTarget | description: Option<Vec<InlineNode>>
ExternalLinkNode   url: String | description: Option<Vec<InlineNode>>
TransclusionNode   url: String | alt: Option<String> | attrs: HashMap<String,String>
RawUrlNode         url: String

Supporting Types

ListItemNode
  symbol:    ListSymbol
  level:     usize
  checkbox:  Option<CheckboxState>
  children:  Vec<InlineNode>
  sublist:   Option<ListNode>

DefinitionItemNode
  term:        Option<Vec<InlineNode>>
  definitions: Vec<Vec<InlineNode>>

TableRowNode   cells: Vec<TableCellNode> | is_header: bool
TableCellNode  children: Vec<InlineNode> | col_span: bool | row_span: bool

LinkTarget
  kind:         LinkKind   -- Wiki|Interwiki|Diary|File|Local|Raw|AnchorOnly
  path:         Option<String>
  wiki_index:   Option<usize>
  wiki_name:    Option<String>
  anchor:       Option<String>
  is_absolute:  bool
  is_directory: bool

ListSymbol    = Dash | Star | Hash | Numeric | NumericParen
              | AlphaParen | AlphaUpperParen | RomanParen | RomanUpperParen
CheckboxState = Empty | Quarter | Half | ThreeQuarters | Done | Rejected
Keyword       = Todo | Done | Started | Fixme | Fixed | Xxx

6.5 Span / Source Location

Every AST node carries a Span:

struct Span {
    start: Position,
    end:   Position,
}

struct Position {
    line:   u32,   -- 0-indexed
    column: u32,   -- 0-indexed, byte offset within line
    offset: usize, -- absolute byte offset from document start
}

Spans are required for LSP diagnostics, semantic tokens, and go-to-definition.

6.6 Lexer Strategy

• Two-pass: block-level pass first, then inline pass within each block
• Token types: syntax-specific (VimwikiToken) — not shared across syntaxes
• TokenStream: eager Vec<Token> wrapped in a TokenStream newtype (changeable to lazy later)
• Operates on Rust chars for correctness; spans stored as byte offsets

6.7 Parser Strategy

• Uses winnow parser combinator library
• Resilient: on malformed input, emits ErrorNode and continues — never fails the whole document
• Inline marker precedence rules documented explicitly in code and tests

6.8 Renderer

• Renderer trait: writer-based (fn render(&self, doc: &DocumentNode, w: &mut dyn Write))
• HtmlRenderer is the first implementation
• Link resolution injected as a callback at construction time
• Template support: minimal {{content}} / {{title}} token substitution
• CSS: linked external stylesheet (matches vimwiki convention)

6.9 LSP Features

Feature	LSP Method
Syntax highlighting	textDocument/semanticTokens/full + /range
Diagnostics (broken links, parse errors)	textDocument/publishDiagnostics
Follow link / go to definition	textDocument/definition
Backlinks	textDocument/references
TOC / outline	textDocument/documentSymbol
Link preview	textDocument/hover
Link autocomplete	textDocument/completion (trigger: [[)
Page rename	workspace/rename
Workspace search	workspace/symbol

6.10 Document Store

DocumentStore: Arc<DashMap<Url, DocumentState>>

DocumentState
  text:     String
  ast:      DocumentNode
  version:  i32

Full re-parse on every didOpen / didChange. Incremental parsing deferred post-v1.

6.11 UTF-16 / Position Encoding

LSP positions negotiated as UTF-8 (positionEncoding = "utf-8") during initialize where the client supports LSP 3.17+. UTF-16 conversion fallback implemented in the LSP layer for older clients. nuwiki-core is always UTF-8 / byte-offset internally.

---

7. Editor Integration

7.1 Plugin Manager Installation

lazy.nvim:

{
  "gffranco/nuwiki",
  build = "lua require('nuwiki').install()",
  ft = { "vimwiki" },
  opts = {},
}

vim-plug:

Plug 'gffranco/nuwiki', { 'do': 'vim -e -s -c "source scripts/download_bin.vim" -c "q"' }

Dein:

call dein#add('gffranco/nuwiki', {
  \ 'build': 'vim -e -s -c "source scripts/download_bin.vim" -c "q"'
  \ })

7.2 Binary Distribution

• Pre-built binaries published as Gitea release assets
• Named: nuwiki-ls-{version}-{target}.tar.gz
• Downloaded at install time by lua/nuwiki/install.lua or scripts/download_bin.vim
• Fallback: cargo build --release if download fails or g:nuwiki_build_from_source = 1
• Binary installed to {plugin_dir}/bin/nuwiki-ls[.exe]

7.3 Editor Detection

" plugin/nuwiki.vim
if has('nvim')
  lua require('nuwiki').setup()
else
  call nuwiki#lsp#start()
endif

7.4 Vim LSP Client Preference Order (Vim only)

1. vim-lsp (matoto/vim-lsp)
2. coc.nvim
3. Error message if neither found

7.5 User Config Schema

require('nuwiki').setup({
  wiki_root      = "~/vimwiki",   -- root directory of the wiki
  file_extension = ".wiki",       -- file extension to associate
  syntax         = "vimwiki",     -- "vimwiki" | "markdown" (future)
  log_level      = "warn",        -- "error"|"warn"|"info"|"debug"
})

7.6 Health Check

:checkhealth nuwiki verifies:

• Binary exists at {plugin_dir}/bin/nuwiki-ls
• Binary is executable and responds to --version
• LSP client is running
• Filetype detection works for .wiki files

---

8. CI/CD

8.1 Runner Setup

Property	Value
System	Gitea Actions (act_runner)
Mode	Docker (jobs run in containers)
Docker access	Host socket mounted (/var/run/docker.sock)
Internet access	Yes

8.2 Workflow Files

.gitea/workflows/ci.yaml — triggers on every push and PR:

• cargo fmt --check
• cargo clippy -- -D warnings
• cargo test --workspace

.gitea/workflows/release.yaml — triggers on v* tag push:

• Cross-compile for all 4 Linux targets using cross
• Package binaries as .tar.gz
• Create Gitea release and upload assets via REST API using RELEASE_TOKEN secret
• Publish nuwiki-core to crates.io using CARGO_REGISTRY_TOKEN secret

8.3 Build Targets

Target	Built by
x86_64-unknown-linux-gnu	CI (cross)
aarch64-unknown-linux-gnu	CI (cross)
x86_64-unknown-linux-musl	CI (cross)
aarch64-unknown-linux-musl	CI (cross)
x86_64-apple-darwin	Manual
aarch64-apple-darwin	Manual
x86_64-pc-windows-msvc	Manual

8.4 Release Secrets

Secret	Purpose
RELEASE_TOKEN	Gitea personal access token for creating releases and uploading assets
CARGO_REGISTRY_TOKEN	crates.io API token for publishing nuwiki-core

8.5 actions/cache Configuration

Requires act_runner's config.yaml to have the cache server host set to the runner host's LAN IP so job containers can reach it.

8.6 Breaking Change Policy

A semver-breaking change (v0.x → v0.x+1 or v1.x → v2.0) is defined as any of:

• AST node type additions, removals, or field changes in nuwiki-core
• SyntaxPlugin or Renderer trait signature changes
• User config schema key removals or type changes
• Removed LSP capabilities

---

9. Vimwiki Syntax Feature Checklist

Typefaces

• Bold: *text*
• Italic: _text_
• Bold italic: _*text*_ / *_text_*
• Strikethrough: ~~text~~
• Inline code: `text`
• Superscript: super^script^
• Subscript: sub,,script,,
• Keywords: TODO DONE STARTED FIXME FIXED XXX

Links

• Plain wikilink: [[Target]]
• Described wikilink: [[Target|Description]]
• Subdirectory wikilink: [[dir/Page]]
• Root-relative wikilink: [[/Page]]
• Filesystem-absolute wikilink: [[//path]]
• Subdirectory link: [[dir/]]
• Interwiki numbered: [[wiki1:Page]]
• Interwiki named: [[wn.Name:Page]]
• Diary link: [[diary:YYYY-MM-DD]]
• Anchor-only link: [[#Anchor]]
• Wikilink with anchor: [[Page#Anchor]]
• Raw URLs: https://… mailto:… ftp://…
• External file link: [[file:path]] / [[local:path]]
• Transclusion: {{URL}} with optional alt and attrs
• Thumbnail link: [[imgURL|{{thumbURL}}]]

Headers

• Levels 1–6: = H1 = … ====== H6 ======
• Centered header (leading whitespace before =)

Lists

• Unordered: - and *
• Ordered: 1. 1) a) A) i) I) #
• Nested and mixed types
• Multi-line items (indentation continuation)
• Definition lists: Term:: Definition
• Checkboxes: [ ] [.] [o] [O] [X] [-]

Tables

• Basic |-delimited table
• Header row separator: |---|
• Column span: >
• Row span: \/
• Inline formatting inside cells

Preformatted Text

• Fenced block: {{{ … }}}
• Optional language/class on opening fence

Mathematical Formulae

• Inline math: $ … $
• Block display math: {{$ … }}$
• Block environment math: {{$%env% … }}$

Blockquotes

• 4-space indent blockquote
• > prefix blockquote

Comments

• Single-line: %% …
• Multi-line: %%+ … +%%

Horizontal Rule

• Four or more dashes: ----

Placeholders

• %title <text>
• %nohtml
• %template <name>
• %date <date>

---

10. Implementation Phases

Phase	Name	Key Output
0	Scaffolding	Compiling empty workspace + CI skeleton
1	Core AST	All node types, spans, Visitor trait
2	Syntax Plugin Interface	Lexer/Parser/SyntaxPlugin traits, SyntaxRegistry
3	Vimwiki Lexer	VimwikiToken, two-pass lexer with spans
4	Vimwiki Parser	Full AST from token stream, error recovery
5	Renderer	Renderer trait + HtmlRenderer
6	LSP Foundation	didOpen/didChange, diagnostics, document symbols
7	Semantic Tokens	AST → LSP semantic token stream
8	Navigation	Definition, references, hover, completion, workspace index
9	Editor Glue	Installable plugin, health check, binary download
10	CI/CD	Automated lint/test/release pipeline

---

11. Pending Decisions

These decisions are required before or during the phase indicated.

#	Decision	Needed by	Options	Notes
P1	License	Phase 0	✅ Dual MIT/Apache-2.0
P2	MSRV	Phase 0	✅ stable-2 (Rust 1.83)
P3	String representation in AST	Phase 1	✅ String (owned)
P4	Visitor pattern	Phase 1	✅ Defined in Phase 1	Open-recursion Visitor trait + walk_* helpers
P5	Minimum Neovim version	Phase 9	0.8 (vim.lsp.start) · 0.11 (vim.lsp.config)	0.8 = broader compat; 0.11 = cleaner Lua API
P6	Minimum Vim version	Phase 9	Vim 8.2 · Vim 9.0 · Vim 9.1	LSP client plugins work best on 9.0+; recommend documenting 9.1 as minimum
P7	Semantic token type mapping	Phase 7	Standard LSP types only · Custom token types	Custom types need client-side highlight group definitions; more flexible but more setup for users
P8	Workspace indexing strategy	Phase 8	Eager on startup · Lazy + background	Eager blocks startup on large wikis; recommend lazy with progress notification
P9	macOS runner	Phase 10	Stay manual · Register macOS runner
Revisit if a Mac is available; no action needed now