phase 7: semantic tokens for syntax highlighting

`textDocument/semanticTokens/full` + `/range` per SPEC §6.9. Token
type scheme settled in P7: custom `vimwiki*` types
(`vimwikiHeading`, `vimwikiBold`, `vimwikiCodeBlock`, `vimwikiKeyword`,
`vimwikiWikilink`, `vimwikiTransclusion`, …) plus `level1`..`level6`
and `centered` modifiers. Default highlight groups for these arrive
in Phase 9.

Pipeline (`semantic_tokens.rs`):

- `legend()` builds the `SemanticTokensLegend` advertised in
  `initialize` capabilities.
- `LineIndex` precomputes byte offsets per line so multi-line span
  splitting and UTF-16 conversion stay linear.
- `collect_tokens` walks the AST. Blocks that are visually distinct
  (heading, preformatted, math-block, comment) emit one token over
  their span; multi-line ones get split per line because the LSP wire
  format can't represent a token that crosses a newline. Container
  blocks (paragraph, list item, table cell, blockquote) recurse into
  inlines instead of wrapping them, so we end up with a non-overlapping
  tile of inline tokens. Headings shadow nested inlines (one heading
  token covers the line, no Bold-inside-Heading).
- `encode` produces the LSP delta-quintuple stream
  `(deltaLine, deltaStart, length, type, mods)`. UTF-16 mode converts
  byte offsets/lengths to code-unit counts on the fly (BMP + surrogate
  pairs).
- `build_data` is the entry point used by both handlers; the `range`
  variant filters tokens by overlap in the same encoding the client
  requested.

Capability declared as `SemanticTokensOptions { full = true,
range = true, legend = … }`. Both handlers reuse the document store.

P7 resolved in SPEC §11 — custom vimwiki types with rationale and a
note that Phase 9 will ship default highlight-group definitions in
`syntax/nuwiki.vim` and the Lua glue.

Tests (21 new): legend integrity, per-construct emission for every
inline + block kind, heading shadowing of inner inlines, multi-line
code/math block splitting, sort order, delta encoding (same line and
crossing lines), UTF-16 length conversion for é, range filter,
empty-doc edge case, end-to-end `build_data` parity.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

crates/nuwiki-lsp/src/lib.rs

@@ -14,6 +14,8 @@
 //!
 //! Re-parses are full per change (incremental parsing deferred post-v1).
 
+pub mod semantic_tokens;
+
 use std::io;
 use std::sync::atomic::{AtomicBool, Ordering};
 use std::sync::Arc;
@@ -25,8 +27,10 @@ use tower_lsp::lsp_types::{
     Diagnostic, DiagnosticSeverity, DidChangeTextDocumentParams, DidCloseTextDocumentParams,
     DidOpenTextDocumentParams, DocumentSymbol, DocumentSymbolParams, DocumentSymbolResponse,
     InitializeParams, InitializeResult, InitializedParams, MessageType, OneOf,
-    PositionEncodingKind, ServerCapabilities, ServerInfo, SymbolKind, TextDocumentSyncCapability,
-    TextDocumentSyncKind, Url,
+    PositionEncodingKind, SemanticTokens, SemanticTokensFullOptions, SemanticTokensOptions,
+    SemanticTokensParams, SemanticTokensRangeParams, SemanticTokensRangeResult,
+    SemanticTokensResult, SemanticTokensServerCapabilities, ServerCapabilities, ServerInfo,
+    SymbolKind, TextDocumentSyncCapability, TextDocumentSyncKind, Url, WorkDoneProgressOptions,
 };
 use tower_lsp::{Client, LanguageServer, LspService, Server};
 
@@ -136,6 +140,16 @@ impl LanguageServer for Backend {
                     TextDocumentSyncKind::FULL,
                 )),
                 document_symbol_provider: Some(OneOf::Left(true)),
+                semantic_tokens_provider: Some(
+                    SemanticTokensServerCapabilities::SemanticTokensOptions(
+                        SemanticTokensOptions {
+                            work_done_progress_options: WorkDoneProgressOptions::default(),
+                            legend: semantic_tokens::legend(),
+                            range: Some(true),
+                            full: Some(SemanticTokensFullOptions::Bool(true)),
+                        },
+                    ),
+                ),
                 ..ServerCapabilities::default()
             },
             server_info: Some(ServerInfo {
@@ -199,6 +213,50 @@ impl LanguageServer for Backend {
             headings_to_symbols(&doc.ast, &doc.text, self.use_utf8.load(Ordering::Relaxed));
         Ok(Some(DocumentSymbolResponse::Nested(symbols)))
     }
+
+    async fn semantic_tokens_full(
+        &self,
+        params: SemanticTokensParams,
+    ) -> LspResult<Option<SemanticTokensResult>> {
+        let uri = &params.text_document.uri;
+        let Some(doc) = self.documents.get(uri) else {
+            return Ok(None);
+        };
+        let utf8 = self.use_utf8.load(Ordering::Relaxed);
+        let data = semantic_tokens::build_data(&doc.ast, &doc.text, utf8, None);
+        Ok(Some(SemanticTokensResult::Tokens(SemanticTokens {
+            result_id: None,
+            data: pack_data(&data),
+        })))
+    }
+
+    async fn semantic_tokens_range(
+        &self,
+        params: SemanticTokensRangeParams,
+    ) -> LspResult<Option<SemanticTokensRangeResult>> {
+        let uri = &params.text_document.uri;
+        let Some(doc) = self.documents.get(uri) else {
+            return Ok(None);
+        };
+        let utf8 = self.use_utf8.load(Ordering::Relaxed);
+        let data = semantic_tokens::build_data(&doc.ast, &doc.text, utf8, Some(params.range));
+        Ok(Some(SemanticTokensRangeResult::Tokens(SemanticTokens {
+            result_id: None,
+            data: pack_data(&data),
+        })))
+    }
+}
+
+fn pack_data(flat: &[u32]) -> Vec<tower_lsp::lsp_types::SemanticToken> {
+    flat.chunks_exact(5)
+        .map(|c| tower_lsp::lsp_types::SemanticToken {
+            delta_line: c[0],
+            delta_start: c[1],
+            length: c[2],
+            token_type: c[3],
+            token_modifiers_bitset: c[4],
+        })
+        .collect()
 }
 
 // ===== Pure helpers =====