phase 7: semantic tokens for syntax highlighting
CI / cargo fmt --check (push) Successful in 30s
CI / cargo clippy (push) Successful in 1m16s
CI / cargo test (push) Successful in 1m20s

`textDocument/semanticTokens/full` + `/range` per SPEC §6.9. Token
type scheme settled in P7: custom `vimwiki*` types
(`vimwikiHeading`, `vimwikiBold`, `vimwikiCodeBlock`, `vimwikiKeyword`,
`vimwikiWikilink`, `vimwikiTransclusion`, …) plus `level1`..`level6`
and `centered` modifiers. Default highlight groups for these arrive
in Phase 9.

Pipeline (`semantic_tokens.rs`):

- `legend()` builds the `SemanticTokensLegend` advertised in
  `initialize` capabilities.
- `LineIndex` precomputes byte offsets per line so multi-line span
  splitting and UTF-16 conversion stay linear.
- `collect_tokens` walks the AST. Blocks that are visually distinct
  (heading, preformatted, math-block, comment) emit one token over
  their span; multi-line ones get split per line because the LSP wire
  format can't represent a token that crosses a newline. Container
  blocks (paragraph, list item, table cell, blockquote) recurse into
  inlines instead of wrapping them, so we end up with a non-overlapping
  tile of inline tokens. Headings shadow nested inlines (one heading
  token covers the line, no Bold-inside-Heading).
- `encode` produces the LSP delta-quintuple stream
  `(deltaLine, deltaStart, length, type, mods)`. UTF-16 mode converts
  byte offsets/lengths to code-unit counts on the fly (BMP + surrogate
  pairs).
- `build_data` is the entry point used by both handlers; the `range`
  variant filters tokens by overlap in the same encoding the client
  requested.

Capability declared as `SemanticTokensOptions { full = true,
range = true, legend = … }`. Both handlers reuse the document store.

P7 resolved in SPEC §11 — custom vimwiki types with rationale and a
note that Phase 9 will ship default highlight-group definitions in
`syntax/nuwiki.vim` and the Lua glue.

Tests (21 new): legend integrity, per-construct emission for every
inline + block kind, heading shadowing of inner inlines, multi-line
code/math block splitting, sort order, delta encoding (same line and
crossing lines), UTF-16 length conversion for é, range filter,
empty-doc edge case, end-to-end `build_data` parity.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-10 21:14:01 +00:00
parent c44db37bab
commit 6efe154f1a
5 changed files with 767 additions and 9 deletions
+2 -2
View File
@@ -1,7 +1,7 @@
# nuwiki — Project Specification
> Last updated: 2026-05-10
> Status: Phase 6 (LSP Foundation) complete — moving to Phase 7 (Semantic Tokens)
> Status: Phase 7 (Semantic Tokens) complete — moving to Phase 8 (Navigation)
---
@@ -590,7 +590,7 @@ These decisions are required before or during the phase indicated.
| ~~P4~~ | ~~**Visitor pattern**~~ | ~~Phase 1~~ | ✅ **Defined in Phase 1** | Open-recursion `Visitor` trait + `walk_*` helpers |
| P5 | **Minimum Neovim version** | Phase 9 | 0.8 (`vim.lsp.start`) · 0.11 (`vim.lsp.config`) | 0.8 = broader compat; 0.11 = cleaner Lua API |
| P6 | **Minimum Vim version** | Phase 9 | Vim 8.2 · Vim 9.0 · Vim 9.1 | LSP client plugins work best on 9.0+; recommend documenting 9.1 as minimum |
| P7 | **Semantic token type mapping** | Phase 7 | Standard LSP types only · Custom token types | Custom types need client-side highlight group definitions; more flexible but more setup for users |
| ~~P7~~ | ~~**Semantic token type mapping**~~ | ~~Phase 7~~ | ✅ **Custom vimwiki-specific token types** | ~20 types (`vimwikiHeading`, `vimwikiBold`, …) + `level1`..`level6` + `centered` modifiers. Phase 9 ships default highlight groups in `syntax/nuwiki.vim` and the Lua glue. |
| P8 | **Workspace indexing strategy** | Phase 8 | Eager on startup · Lazy + background | Eager blocks startup on large wikis; recommend lazy with progress notification |
| P9 | **macOS runner** | Phase 10 | Stay manual · Register macOS runner
| Revisit if a Mac is available; no action needed now |