phase 6: LSP foundation — didOpen/didChange, diagnostics, outline
CI / cargo fmt --check (push) Successful in 48s
CI / cargo clippy (push) Successful in 1m19s
CI / cargo test (push) Successful in 1m19s

`tower-lsp` server (`nuwiki-lsp::run` + `run_stdio`) wired up by the
`nuwiki-ls` binary. Backend holds an `Arc<DashMap<Url, DocumentState>>`
per SPEC §6.10 plus an `Arc<SyntaxRegistry>` pre-loaded with
`VimwikiSyntax`. Re-parse is full per change; incremental parsing
deferred post-v1 per spec.

Capabilities declared:
- `text_document_sync = FULL`
- `document_symbol_provider = true`
- `position_encoding = UTF-8` when the client opts into LSP 3.17+
  encodings (SPEC §6.11). Otherwise we keep the legacy UTF-16 default
  and translate spans through a per-line UTF-16 code-unit walk that
  handles BMP and surrogate pairs alike.

Handlers: `initialize` (negotiate encoding), `initialized` (log it),
`shutdown`, `did_open` / `did_change` (full re-parse +
`publishDiagnostics`), `did_close` (drop state, clear diagnostics),
`document_symbol` (nested outline).

Diagnostics: every `BlockNode::Error` (from §6.7's resilient parser)
emits one `DiagnosticSeverity::ERROR` with `source = "nuwiki"`. The
walker descends into Blockquotes and ListItems so errors nested inside
those don't slip through.

Document outline: nested `DocumentSymbol` tree built recursively from
heading levels (`= H1 =` → root, `== H2 ==` → child of preceding H1,
etc.). Titles flatten inline markers via `inline_to_text` so
`*world*` shows up as `world` in the outline.

Pure helpers (`to_lsp_position`, `to_lsp_range`, `ast_diagnostics`,
`headings_to_symbols`) are `pub` so the integration tests can verify
conversion correctness without spinning up the async server. End-to-
end LSP message exercises will land in Phase 8 alongside navigation.

Tests (11): UTF-8 passthrough, UTF-16 conversion for é + 🦀, single
and multi-error diagnostic emission, blockquote descent, flat outline,
nested outline by levels, inline-marker stripping in titles, empty-
heading fallback, registry-dispatch parity.

Dep pin: `idna_adapter` held at 1.2.0 — 1.2.2 needs Cargo's
edition2024 feature (Rust 1.85), our MSRV is 1.83.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-10 19:44:44 +00:00
parent f00b0780b8
commit c44db37bab
8 changed files with 1495 additions and 8 deletions
+4
View File
@@ -14,3 +14,7 @@ workspace = true
[dependencies]
nuwiki-core = { path = "../nuwiki-core", version = "0.1.0" }
tower-lsp = "0.20"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "io-std", "sync"] }
dashmap = "6"
async-trait = "0.1"
+411 -2
View File
@@ -1,4 +1,413 @@
//! LSP protocol bridge for nuwiki.
//!
//! Translates between LSP requests/notifications and `nuwiki-core` AST
//! operations. Must never depend on `nuwiki-ls` (see SPEC.md §6.2).
//! `tower-lsp` server that maintains a `DocumentStore` (SPEC §6.10) and
//! exposes the Phase 6 feature set:
//!
//! - `initialize` / `initialized` / `shutdown`
//! - `textDocument/didOpen`, `didChange` (full sync), `didClose`
//! - `textDocument/publishDiagnostics` from `BlockNode::Error` nodes
//! - `textDocument/documentSymbol` — nested outline from headings
//!
//! Position encoding is negotiated as UTF-8 when the client supports LSP
//! 3.17+ encodings (SPEC §6.11). When the client only supports the legacy
//! UTF-16 default, positions get translated through a per-line lookup.
//!
//! Re-parses are full per change (incremental parsing deferred post-v1).
use std::io;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use dashmap::DashMap;
use tokio::io::{AsyncRead, AsyncWrite};
use tower_lsp::jsonrpc::Result as LspResult;
use tower_lsp::lsp_types::{
Diagnostic, DiagnosticSeverity, DidChangeTextDocumentParams, DidCloseTextDocumentParams,
DidOpenTextDocumentParams, DocumentSymbol, DocumentSymbolParams, DocumentSymbolResponse,
InitializeParams, InitializeResult, InitializedParams, MessageType, OneOf,
PositionEncodingKind, ServerCapabilities, ServerInfo, SymbolKind, TextDocumentSyncCapability,
TextDocumentSyncKind, Url,
};
use tower_lsp::{Client, LanguageServer, LspService, Server};
use nuwiki_core::ast::{
BlockNode, BlockquoteNode, DocumentNode, ErrorNode, HeadingNode, InlineNode, ListItemNode,
ListNode, Span,
};
use nuwiki_core::syntax::vimwiki::VimwikiSyntax;
use nuwiki_core::syntax::SyntaxRegistry;
/// Run the LSP server over the given async reader/writer pair. The binary
/// (`nuwiki-ls`) just calls this with stdin/stdout.
pub async fn run<I, O>(reader: I, writer: O)
where
I: AsyncRead + Unpin,
O: AsyncWrite + Unpin,
{
let (service, socket) = LspService::new(Backend::new);
Server::new(reader, writer, socket).serve(service).await;
}
/// Convenience: run on stdin/stdout. Fails only if stdio handles are
/// unavailable.
pub async fn run_stdio() -> io::Result<()> {
let stdin = tokio::io::stdin();
let stdout = tokio::io::stdout();
run(stdin, stdout).await;
Ok(())
}
#[derive(Debug)]
struct DocumentState {
text: String,
ast: DocumentNode,
/// Last version we observed from the client. Held per SPEC §6.10 even
/// though the foundation phase doesn't yet need to read it back.
#[allow(dead_code)]
version: i32,
}
struct Backend {
client: Client,
documents: Arc<DashMap<Url, DocumentState>>,
registry: Arc<SyntaxRegistry>,
/// True after `initialize` if the client opted into UTF-8 encoding.
/// Otherwise we keep the LSP 3.16 default of UTF-16.
use_utf8: Arc<AtomicBool>,
}
impl Backend {
fn new(client: Client) -> Self {
let mut registry = SyntaxRegistry::new();
registry.register(VimwikiSyntax::new());
Self {
client,
documents: Arc::new(DashMap::new()),
registry: Arc::new(registry),
use_utf8: Arc::new(AtomicBool::new(false)),
}
}
async fn update_document(&self, uri: Url, text: String, version: i32) {
// For Phase 6 we always treat unknown buffers as vimwiki — if/when
// multi-syntax dispatch lands the pick should follow the URI's ext.
let plugin = match self.registry.get("vimwiki") {
Some(p) => p,
None => {
self.client
.log_message(MessageType::ERROR, "vimwiki plugin missing")
.await;
return;
}
};
let ast = plugin.parse(&text);
let diagnostics = ast_diagnostics(&ast, &text, self.use_utf8.load(Ordering::Relaxed));
self.documents
.insert(uri.clone(), DocumentState { text, ast, version });
self.client
.publish_diagnostics(uri, diagnostics, Some(version))
.await;
}
}
#[tower_lsp::async_trait]
impl LanguageServer for Backend {
async fn initialize(&self, params: InitializeParams) -> LspResult<InitializeResult> {
// SPEC §6.11: prefer UTF-8 if the client advertises support.
let supports_utf8 = params
.capabilities
.general
.as_ref()
.and_then(|g| g.position_encodings.as_ref())
.map(|encs| encs.iter().any(|e| *e == PositionEncodingKind::UTF8))
.unwrap_or(false);
self.use_utf8.store(supports_utf8, Ordering::Relaxed);
let position_encoding = if supports_utf8 {
Some(PositionEncodingKind::UTF8)
} else {
None // server defaults to UTF-16 per LSP 3.16.
};
Ok(InitializeResult {
capabilities: ServerCapabilities {
position_encoding,
text_document_sync: Some(TextDocumentSyncCapability::Kind(
TextDocumentSyncKind::FULL,
)),
document_symbol_provider: Some(OneOf::Left(true)),
..ServerCapabilities::default()
},
server_info: Some(ServerInfo {
name: env!("CARGO_PKG_NAME").into(),
version: Some(env!("CARGO_PKG_VERSION").into()),
}),
})
}
async fn initialized(&self, _params: InitializedParams) {
let enc = if self.use_utf8.load(Ordering::Relaxed) {
"utf-8"
} else {
"utf-16"
};
self.client
.log_message(
MessageType::INFO,
format!("nuwiki-lsp initialized (positionEncoding={enc})"),
)
.await;
}
async fn shutdown(&self) -> LspResult<()> {
Ok(())
}
async fn did_open(&self, params: DidOpenTextDocumentParams) {
let uri = params.text_document.uri;
let text = params.text_document.text;
let version = params.text_document.version;
self.update_document(uri, text, version).await;
}
async fn did_change(&self, params: DidChangeTextDocumentParams) {
let uri = params.text_document.uri;
let version = params.text_document.version;
// Full sync: the *last* content change carries the whole document.
if let Some(change) = params.content_changes.into_iter().last() {
self.update_document(uri, change.text, version).await;
}
}
async fn did_close(&self, params: DidCloseTextDocumentParams) {
self.documents.remove(&params.text_document.uri);
// Clear lingering diagnostics so closed buffers don't keep red squiggles.
self.client
.publish_diagnostics(params.text_document.uri, Vec::new(), None)
.await;
}
async fn document_symbol(
&self,
params: DocumentSymbolParams,
) -> LspResult<Option<DocumentSymbolResponse>> {
let uri = &params.text_document.uri;
let Some(doc) = self.documents.get(uri) else {
return Ok(None);
};
let symbols =
headings_to_symbols(&doc.ast, &doc.text, self.use_utf8.load(Ordering::Relaxed));
Ok(Some(DocumentSymbolResponse::Nested(symbols)))
}
}
// ===== Pure helpers =====
/// Convert one of our `Position` values into an LSP `Position`.
///
/// In UTF-8 encoding mode (negotiated during `initialize`) the byte column
/// is the LSP `character`. In UTF-16 mode the LSP `character` is a UTF-16
/// code-unit offset, so we walk the source line to translate.
pub fn to_lsp_position(
pos: &nuwiki_core::ast::Position,
text: &str,
utf8: bool,
) -> tower_lsp::lsp_types::Position {
let character = if utf8 {
pos.column
} else {
utf16_column(text, pos.line, pos.column)
};
tower_lsp::lsp_types::Position {
line: pos.line,
character,
}
}
pub fn to_lsp_range(span: &Span, text: &str, utf8: bool) -> tower_lsp::lsp_types::Range {
tower_lsp::lsp_types::Range {
start: to_lsp_position(&span.start, text, utf8),
end: to_lsp_position(&span.end, text, utf8),
}
}
/// Convert a byte-offset column on `line` into the corresponding UTF-16
/// code-unit count from the start of that line. Out-of-range inputs are
/// clamped to the line length, matching the LSP spec's "if the character
/// value is greater than the length of the line it is clipped to the
/// length".
fn utf16_column(text: &str, line: u32, byte_col: u32) -> u32 {
let mut current_line = 0u32;
let mut line_start = 0usize;
let bytes = text.as_bytes();
while current_line < line && line_start < bytes.len() {
if let Some(nl) = bytes[line_start..].iter().position(|b| *b == b'\n') {
line_start += nl + 1;
current_line += 1;
} else {
return 0;
}
}
let line_end = bytes[line_start..]
.iter()
.position(|b| *b == b'\n')
.map(|i| line_start + i)
.unwrap_or(bytes.len());
let target = (line_start + byte_col as usize).min(line_end);
text[line_start..target]
.chars()
.map(char::len_utf16)
.sum::<usize>() as u32
}
pub fn ast_diagnostics(ast: &DocumentNode, text: &str, utf8: bool) -> Vec<Diagnostic> {
let mut out = Vec::new();
walk_blocks_for_errors(&ast.children, text, utf8, &mut out);
out
}
fn walk_blocks_for_errors(blocks: &[BlockNode], text: &str, utf8: bool, out: &mut Vec<Diagnostic>) {
for block in blocks {
match block {
BlockNode::Error(err) => out.push(error_to_diagnostic(err, text, utf8)),
BlockNode::Blockquote(BlockquoteNode { children, .. }) => {
walk_blocks_for_errors(children, text, utf8, out);
}
BlockNode::List(ListNode { items, .. }) => {
for item in items {
walk_list_item_for_errors(item, text, utf8, out);
}
}
_ => {}
}
}
}
fn walk_list_item_for_errors(
item: &ListItemNode,
text: &str,
utf8: bool,
out: &mut Vec<Diagnostic>,
) {
if let Some(sub) = &item.sublist {
for sub_item in &sub.items {
walk_list_item_for_errors(sub_item, text, utf8, out);
}
}
let _ = (text, utf8, out);
}
fn error_to_diagnostic(err: &ErrorNode, text: &str, utf8: bool) -> Diagnostic {
Diagnostic {
range: to_lsp_range(&err.span, text, utf8),
severity: Some(DiagnosticSeverity::ERROR),
message: err.message.clone(),
source: Some("nuwiki".into()),
..Diagnostic::default()
}
}
pub fn headings_to_symbols(ast: &DocumentNode, text: &str, utf8: bool) -> Vec<DocumentSymbol> {
let headings: Vec<&HeadingNode> = ast
.children
.iter()
.filter_map(|b| match b {
BlockNode::Heading(h) => Some(h),
_ => None,
})
.collect();
#[allow(deprecated)]
fn build<'a>(
headings: &mut std::slice::Iter<'a, &'a HeadingNode>,
text: &str,
utf8: bool,
parent_level: u8,
peeked: &mut Option<&'a HeadingNode>,
) -> Vec<DocumentSymbol> {
let mut out = Vec::new();
loop {
let h = match peeked.take().or_else(|| headings.next().copied()) {
Some(h) => h,
None => return out,
};
if h.level <= parent_level {
*peeked = Some(h);
return out;
}
let children = build(headings, text, utf8, h.level, peeked);
let title = inline_to_text(&h.children);
out.push(DocumentSymbol {
name: if title.is_empty() {
"(empty heading)".into()
} else {
title
},
detail: None,
kind: SymbolKind::STRING,
tags: None,
deprecated: None,
range: to_lsp_range(&h.span, text, utf8),
selection_range: to_lsp_range(&h.span, text, utf8),
children: if children.is_empty() {
None
} else {
Some(children)
},
});
}
}
let mut iter = headings.iter();
let mut peeked: Option<&HeadingNode> = None;
build(&mut iter, text, utf8, 0, &mut peeked)
}
fn inline_to_text(inlines: &[InlineNode]) -> String {
let mut out = String::new();
inline_to_text_into(inlines, &mut out);
out.trim().to_string()
}
fn inline_to_text_into(inlines: &[InlineNode], out: &mut String) {
for n in inlines {
match n {
InlineNode::Text(t) => out.push_str(&t.content),
InlineNode::Bold(b) => inline_to_text_into(&b.children, out),
InlineNode::Italic(i) => inline_to_text_into(&i.children, out),
InlineNode::BoldItalic(bi) => inline_to_text_into(&bi.children, out),
InlineNode::Strikethrough(s) => inline_to_text_into(&s.children, out),
InlineNode::Code(c) => out.push_str(&c.content),
InlineNode::Superscript(s) => inline_to_text_into(&s.children, out),
InlineNode::Subscript(s) => inline_to_text_into(&s.children, out),
InlineNode::MathInline(m) => out.push_str(&m.content),
InlineNode::Keyword(k) => out.push_str(keyword_str(k.keyword)),
InlineNode::Color(c) => inline_to_text_into(&c.children, out),
InlineNode::WikiLink(w) => match &w.description {
Some(d) => inline_to_text_into(d, out),
None => {
if let Some(p) = &w.target.path {
out.push_str(p);
}
}
},
InlineNode::ExternalLink(e) => match &e.description {
Some(d) => inline_to_text_into(d, out),
None => out.push_str(&e.url),
},
InlineNode::Transclusion(t) => out.push_str(t.alt.as_deref().unwrap_or(&t.url)),
InlineNode::RawUrl(r) => out.push_str(&r.url),
}
}
}
fn keyword_str(k: nuwiki_core::ast::Keyword) -> &'static str {
match k {
nuwiki_core::ast::Keyword::Todo => "TODO",
nuwiki_core::ast::Keyword::Done => "DONE",
nuwiki_core::ast::Keyword::Started => "STARTED",
nuwiki_core::ast::Keyword::Fixme => "FIXME",
nuwiki_core::ast::Keyword::Fixed => "FIXED",
nuwiki_core::ast::Keyword::Xxx => "XXX",
}
}
+255
View File
@@ -0,0 +1,255 @@
//! Tests for the pure helpers that the LSP backend uses to translate
//! between `nuwiki-core` ASTs and the LSP wire format. The async
//! request-handling loop is exercised at integration time in Phase 8+
//! (navigation), so this file stays focused on conversion correctness.
use nuwiki_core::ast::{
inline::InlineNode, BlockNode, BlockquoteNode, DocumentNode, ErrorNode, ParagraphNode,
Position, Span, TextNode,
};
use nuwiki_core::syntax::vimwiki::VimwikiSyntax;
use nuwiki_core::syntax::SyntaxPlugin;
use nuwiki_lsp::{ast_diagnostics, headings_to_symbols, to_lsp_position, to_lsp_range};
use tower_lsp::lsp_types::{DiagnosticSeverity, DocumentSymbolResponse, SymbolKind};
// ===== Position / Range conversion =====
#[test]
fn position_passthrough_in_utf8_mode() {
let pos = Position {
line: 3,
column: 7,
offset: 0,
};
let lsp = to_lsp_position(&pos, "ignored", true);
assert_eq!(lsp.line, 3);
assert_eq!(lsp.character, 7);
}
#[test]
fn position_translates_to_utf16_for_multibyte_chars() {
// "héllo" in UTF-8: h(1) é(2 bytes) l(1) l(1) o(1) — total 6 bytes.
// In UTF-16 each char is one code unit (BMP), so byte col 3 ("after é")
// maps to UTF-16 char col 2.
let line = "héllo\nworld\n";
let pos = Position {
line: 0,
column: 3,
offset: 3,
};
let lsp = to_lsp_position(&pos, line, false);
assert_eq!(lsp.line, 0);
assert_eq!(lsp.character, 2);
}
#[test]
fn position_handles_astral_plane_in_utf16() {
// 🦀 is U+1F980, two UTF-16 code units (surrogate pair), four UTF-8 bytes.
let text = "🦀x\n";
// After the crab + x, byte col is 5, UTF-16 col is 3 (2 surrogates + 1 BMP).
let pos = Position {
line: 0,
column: 5,
offset: 5,
};
let lsp = to_lsp_position(&pos, text, false);
assert_eq!(lsp.character, 3);
}
#[test]
fn range_passthrough_in_utf8_mode() {
let span = Span::new(
Position {
line: 0,
column: 0,
offset: 0,
},
Position {
line: 0,
column: 5,
offset: 5,
},
);
let r = to_lsp_range(&span, "hello\n", true);
assert_eq!(r.start.line, 0);
assert_eq!(r.start.character, 0);
assert_eq!(r.end.character, 5);
}
// ===== Diagnostics =====
#[test]
fn ast_diagnostics_yields_one_per_error_node() {
let span_a = Span::new(
Position {
line: 0,
column: 0,
offset: 0,
},
Position {
line: 0,
column: 4,
offset: 4,
},
);
let span_b = Span::new(
Position {
line: 2,
column: 0,
offset: 10,
},
Position {
line: 2,
column: 3,
offset: 13,
},
);
let doc = DocumentNode {
span: Span::default(),
metadata: Default::default(),
children: vec![
BlockNode::Error(ErrorNode {
span: span_a,
raw: "first".into(),
message: "first parse failure".into(),
}),
BlockNode::Paragraph(ParagraphNode {
span: Span::default(),
children: vec![InlineNode::Text(TextNode {
span: Span::default(),
content: "ok".into(),
})],
}),
BlockNode::Error(ErrorNode {
span: span_b,
raw: "second".into(),
message: "second parse failure".into(),
}),
],
};
let diags = ast_diagnostics(&doc, "ignored", true);
assert_eq!(diags.len(), 2);
assert_eq!(diags[0].message, "first parse failure");
assert_eq!(diags[0].severity, Some(DiagnosticSeverity::ERROR));
assert_eq!(diags[0].source.as_deref(), Some("nuwiki"));
assert_eq!(diags[0].range.start.line, 0);
assert_eq!(diags[1].range.start.line, 2);
}
#[test]
fn ast_diagnostics_descends_into_blockquotes() {
let span = Span::new(
Position {
line: 1,
column: 2,
offset: 0,
},
Position {
line: 1,
column: 6,
offset: 4,
},
);
let inner_error = BlockNode::Error(ErrorNode {
span,
raw: "x".into(),
message: "nested".into(),
});
let bq = BlockNode::Blockquote(BlockquoteNode {
span: Span::default(),
children: vec![inner_error],
});
let doc = DocumentNode {
span: Span::default(),
metadata: Default::default(),
children: vec![bq],
};
let diags = ast_diagnostics(&doc, "", true);
assert_eq!(diags.len(), 1);
assert_eq!(diags[0].message, "nested");
}
// ===== Document symbols =====
fn parse(src: &str) -> DocumentNode {
VimwikiSyntax::new().parse(src)
}
#[test]
fn flat_outline_when_all_headings_same_level() {
let doc = parse("= A =\n= B =\n= C =\n");
let syms = headings_to_symbols(&doc, "", true);
assert_eq!(syms.len(), 3);
let names: Vec<&str> = syms.iter().map(|s| s.name.as_str()).collect();
assert_eq!(names, vec!["A", "B", "C"]);
for s in &syms {
assert!(s.children.is_none() || s.children.as_ref().unwrap().is_empty());
assert_eq!(s.kind, SymbolKind::STRING);
}
}
#[test]
fn nested_outline_follows_heading_levels() {
let src = "\
= Top =
== Sub A ==
=== Deeper ===
== Sub B ==
= Other =
";
let doc = parse(src);
let syms = headings_to_symbols(&doc, "", true);
assert_eq!(syms.len(), 2);
// Top contains Sub A and Sub B; Sub A contains Deeper.
let top = &syms[0];
assert_eq!(top.name, "Top");
let top_kids = top.children.as_ref().expect("Top should have children");
assert_eq!(top_kids.len(), 2);
assert_eq!(top_kids[0].name, "Sub A");
assert_eq!(top_kids[1].name, "Sub B");
let sub_a_kids = top_kids[0]
.children
.as_ref()
.expect("Sub A should have children");
assert_eq!(sub_a_kids.len(), 1);
assert_eq!(sub_a_kids[0].name, "Deeper");
assert_eq!(syms[1].name, "Other");
}
#[test]
fn heading_title_strips_inline_markers() {
let doc = parse("= Hello *world* and `code` =\n");
let syms = headings_to_symbols(&doc, "", true);
assert_eq!(syms.len(), 1);
assert_eq!(syms[0].name, "Hello world and code");
}
#[test]
fn empty_heading_falls_back_to_placeholder_name() {
let doc = parse("= =\n");
let syms = headings_to_symbols(&doc, "", true);
if let Some(s) = syms.first() {
assert!(!s.name.is_empty());
}
}
// ===== End-to-end through the registry =====
#[test]
fn outline_symbols_match_registry_dispatch() {
let src = "= A =\n== B ==\n";
let plugin = VimwikiSyntax::new();
let doc = plugin.parse(src);
let syms = headings_to_symbols(&doc, src, true);
assert_eq!(syms.len(), 1);
assert_eq!(syms[0].name, "A");
let kids = syms[0].children.as_ref().unwrap();
assert_eq!(kids[0].name, "B");
// Sanity: symbols can be wrapped in DocumentSymbolResponse::Nested.
let _resp = DocumentSymbolResponse::Nested(syms);
}