e530b3d62354f7fdf2c1e2b26e7bfb5b1f5a5425
55 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 gffranco
|
8b29d4e9b9 |
phase 12: tags — lexer, AST, index, anchor resolution, ws/symbol
Vimwiki `:tag1:tag2:` syntax across the full stack. P11 resolved as
strict-vimwiki — no markdown `#tag` alias.
AST (nuwiki-core):
- `TagScope { File, Heading(usize), Standalone }` mirrors vimwiki's
three placement-determined meanings: lines 0-1 → File; one or two
lines below the most recent heading → Heading(idx); otherwise
Standalone.
- `TagNode { span, tags: Vec<String>, scope }` + `BlockNode::Tag`.
- `PageMetadata.tags: Vec<String>` accumulates the file-scope tag
names — convenience projection of the Tag blocks whose scope is
File. New field is additive (Default derive) so existing consumers
using `..Default::default()` keep working.
- `Visitor::visit_tag` lands with a default body and `walk_block`
dispatches the new arm.
Lexer (`syntax/vimwiki/lexer.rs`):
- `VimwikiTokenKind::Tag(Vec<String>)` block-level token.
- `try_lex_tag` runs in the normal-line dispatcher right after heading
recognition; demands the *whole trimmed line* match `:name:name…:`,
with non-empty whitespace-free names between adjacent colons. Empty
segments (`::tag::`), whitespace inside a name, or single-colon
lines are all rejected. Doesn't fight `Term::` (definition-list
marker) because that pattern requires text before the `::`.
- Indentation is tolerated; the span covers the trimmed run.
Parser (`syntax/vimwiki/parser.rs`):
- `ParseState` gains `headings_seen` + `last_heading_line`.
- `parse_heading` bumps both on every closed heading so the tag
parser can detect the heading window.
- `parse_tag` reads the Tag token, computes the scope via
`scope_for_tag(tag_line)`, eats the trailing newline, and emits the
TagNode.
- `parse_document` post-processes each block: when a Tag with
`scope == File` is emitted, its names are pushed (de-duped) into
`metadata.tags`.
Renderer (`render/html.rs`):
- `render_tag` emits `<div class="tags"><span class="tag"
id="tag-name">name</span>...</div>`. The `id` is what makes
`[[Page#tag-name]]` jump to the rendered tag.
Semantic tokens (`semantic_tokens.rs`):
- New `TOKEN_TAG = 19` / `vimwikiTag` so editors highlight tag lines
distinctly. `emit_block` adds the Tag arm.
Workspace index (`index.rs`):
- `IndexedPage.tags: Vec<TagInfo>` records each tag occurrence on a
page with its span + scope.
- `WorkspaceIndex.tags_by_name: HashMap<String, Vec<TagOccurrence>>`
is the reverse map for cross-page anchor resolution + the Phase 13
`nuwiki.tags.search` command.
- `upsert` populates both; `remove` scrubs the reverse map and drops
empty entries; `tags_for(name)` is the public lookup.
LSP handlers (`lib.rs`):
- `heading_range_in` falls back to tag matching when the anchor
isn't a heading slug — so `[[Page#some-tag]]` lands on the matching
TagNode. Heading anchors win over tag anchors when both match (the
vimwiki precedence).
- `workspace/symbol` includes tags alongside headings, named `:tag:`
and kinded `PROPERTY` so editor UIs can distinguish them.
Tests (20 new):
- Syntax (15 in `vimwiki_tags.rs`):
lexer recognition + rejection of `::tag::`, whitespace-in-name,
Term::-conflict; parser scope paths (file, heading-1-below,
heading-2-below, standalone, no-heading, line-1 priority);
metadata dedup; renderer HTML + escaping.
- LSP index (5 in `tags_index_and_lsp.rs`):
per-page tag population, cross-page `tags_by_name` aggregation,
re-upsert replaces stale tags, `remove` cleans the reverse map,
heading-scope tag indexed but not in file metadata.
All 191 v1.0 + Phase 11 tests still pass — backward compatible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
gffranco
|
ea574f23f8 |
phase 11: plumbing prereqs — Wiki, edits, config, snapshot, diags
Cross-cutting infrastructure that every v1.1 phase depends on. Lands as a numbered phase before any user-visible feature so Phases 12–19 stay mechanical. 5 pieces: - `config.rs` — `Config` + `WikiConfig` + `DiagnosticConfig` + `LinkSeverity`. Serde-deserialised from `initialization_options`. v1.0 single-wiki shape (`wiki_root` + `file_extension` + `syntax`) desugars to one-entry `wikis = [...]` per P16. `~/...` expansion without pulling in `dirs`. Fallbacks: workspace_folders[0] → deprecated root_uri → empty list. - `wiki.rs` — `Wiki` aggregate (id + WikiConfig + Arc<RwLock<Index>>). `build_wikis(&[WikiConfig])` materialises the list; `resolve_uri_to_wiki` picks the longest-prefix root match so nested wikis route correctly. Single-entry today, multi-entry once Phase 18 lands. - `edits.rs` — `WorkspaceEditBuilder` + `text_edit_replace/insert/delete` + `op_rename/delete/create`. Handles the UTF-8/UTF-16 conversion at the span boundary so each Phase 13+ command stays a 3-liner. Builder picks `documentChanges` when file ops are present (LSP requires it), `changes` map otherwise (works for LSP 3.15- clients). File ops precede content edits in the output so created/renamed files exist when edits apply. `DeleteFile` is built without `annotation_id` — lsp-types 0.94.x ships it that way; later versions added the field. - `Backend` refactor — `index: Arc<RwLock<WorkspaceIndex>>` replaced by `wikis: Arc<RwLock<Vec<Wiki>>>` + `config: Arc<RwLock<Config>>`. New `wiki(id)` / `default_wiki()` / `wiki_for_uri(uri)` helpers. Every v1.0 handler (definition, references, hover, completion, workspace/symbol, update_document) goes through `wiki_for_uri`. `initialize` builds wikis from `Config::from_init_params`; `initialized` spawns one indexer task per wiki with the existing `window/workDoneProgress` wrapper. New `did_change_configuration` handler re-applies settings (rebuild semantics revisited in Phase 18). The old `workspace_root_from_params` helper is gone. - `read_or_open` + `DocSnapshot` (scaffolded, `#[allow(dead_code)]` until Phase 13's `workspace/rename` consumes it). Returns the live document when held in the documents map, otherwise async-reads from disk and re-parses so cross-document operations get accurate text and spans instead of trusting stale `WorkspaceIndex` data. - `collect_diagnostics` composable collector. Same `ErrorNode` walk today; takes `Option<&WorkspaceIndex>` + `page_name` + `&DiagnosticConfig` so Phase 15 can drop a link-health source in without further refactor. `ast_diagnostics` kept as a thin wrapper for the v1.0 test surface. New deps: `serde` + `serde_json` (already transitive through tower-lsp; elevated to direct deps for clarity and stability). `tokio` gains the `fs` feature for `read_or_open`'s async disk read. Tests (19 new in `phase11_plumbing.rs`): - Edits: UTF-8 passthrough + UTF-16 conversion (`héllo`), insert/delete shape, builder mode selection, file-op accumulation, `op_create` round-trip - Config: defaults, v1.0 desugar, explicit list overrides legacy, empty JSON, invalid JSON preserves prior state - Wiki: sequential id assignment, longest-prefix resolution, no-match, `Wiki::contains` - Diagnostics: error-node composition, link-health stub returns empty in Phase 11 All 172 v1.0 tests still pass — backward compatible. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
gffranco
|
4d461d2425 |
phase 8: navigation — definition, refs, hover, completion, ws/symbol
P8 resolved (SPEC §11): lazy + background indexing with progress. Initial scan runs in a tokio task spawned from `initialized`; the server stays responsive immediately and nav features answer with partial data until the index is complete. WorkspaceIndex (`index.rs`): - Per-URI `IndexedPage` keeps name, title, headings, outgoing links. - Reverse maps: `pages_by_name`, `backlinks: HashMap<page, …>`. - `upsert` scrubs stale entries first so re-parses don't leak. - `resolve` handles Wiki + AnchorOnly link kinds against the index; other kinds (Interwiki/Diary/File/Local/Raw) fall through. - `slugify` for anchors (lowercases, collapses runs, drops trailing dashes, preserves Unicode), `page_name_from_uri` for workspace- rooted names, `walk_wiki_files` for the background scanner. Backend wiring (`lib.rs`): - `initialize` captures workspace root from `workspaceFolders` or the deprecated `root_uri`; advertises definition/references/hover/ completion (trigger `[`)/workspace_symbol providers. - `initialized` spawns the indexer, wrapped in `window/workDoneProgress` begin/end notifications. - `did_open`/`did_change` synchronously update the index alongside publishing diagnostics; `did_close` keeps the indexed projection (file may still exist on disk and other pages link to it) and only clears diagnostics + drops the live document state. Position lookup (`nav.rs`): - `lsp_to_byte_pos` translates between LSP encoding (UTF-8 or UTF-16) and our byte columns. - `find_inline_at` descends blocks → inlines → inline containers, returning the most specific inline whose span covers the cursor. Handlers: - definition: wikilinks resolve to a target URI; anchored links return the range at the matching heading. - references: backlinks of the cursor's link target, or of the current page when the cursor isn't on a link. Ranges fall back to byte-column LSP positions when the source doc isn't open. - hover: markdown preview with page title + outline (first 8 headings), or a "not in index" fallback. - completion: fires only inside an unterminated `[[ … ` on the current line; suggests every indexed page name. - workspace/symbol: case-insensitive substring over every indexed heading. Tests (20 new): slugify (basic / punctuation / Unicode / trailing dash), page-name derivation under various roots, upsert / replace / remove, resolve for Wiki + AnchorOnly, sorted `page_names`, UTF-8 passthrough + UTF-16 conversion, position lookup hits and misses. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
gffranco
|
6efe154f1a |
phase 7: semantic tokens for syntax highlighting
`textDocument/semanticTokens/full` + `/range` per SPEC §6.9. Token
type scheme settled in P7: custom `vimwiki*` types
(`vimwikiHeading`, `vimwikiBold`, `vimwikiCodeBlock`, `vimwikiKeyword`,
`vimwikiWikilink`, `vimwikiTransclusion`, …) plus `level1`..`level6`
and `centered` modifiers. Default highlight groups for these arrive
in Phase 9.
Pipeline (`semantic_tokens.rs`):
- `legend()` builds the `SemanticTokensLegend` advertised in
`initialize` capabilities.
- `LineIndex` precomputes byte offsets per line so multi-line span
splitting and UTF-16 conversion stay linear.
- `collect_tokens` walks the AST. Blocks that are visually distinct
(heading, preformatted, math-block, comment) emit one token over
their span; multi-line ones get split per line because the LSP wire
format can't represent a token that crosses a newline. Container
blocks (paragraph, list item, table cell, blockquote) recurse into
inlines instead of wrapping them, so we end up with a non-overlapping
tile of inline tokens. Headings shadow nested inlines (one heading
token covers the line, no Bold-inside-Heading).
- `encode` produces the LSP delta-quintuple stream
`(deltaLine, deltaStart, length, type, mods)`. UTF-16 mode converts
byte offsets/lengths to code-unit counts on the fly (BMP + surrogate
pairs).
- `build_data` is the entry point used by both handlers; the `range`
variant filters tokens by overlap in the same encoding the client
requested.
Capability declared as `SemanticTokensOptions { full = true,
range = true, legend = … }`. Both handlers reuse the document store.
P7 resolved in SPEC §11 — custom vimwiki types with rationale and a
note that Phase 9 will ship default highlight-group definitions in
`syntax/nuwiki.vim` and the Lua glue.
Tests (21 new): legend integrity, per-construct emission for every
inline + block kind, heading shadowing of inner inlines, multi-line
code/math block splitting, sort order, delta encoding (same line and
crossing lines), UTF-16 length conversion for é, range filter,
empty-doc edge case, end-to-end `build_data` parity.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
gffranco
|
c44db37bab |
phase 6: LSP foundation — didOpen/didChange, diagnostics, outline
`tower-lsp` server (`nuwiki-lsp::run` + `run_stdio`) wired up by the `nuwiki-ls` binary. Backend holds an `Arc<DashMap<Url, DocumentState>>` per SPEC §6.10 plus an `Arc<SyntaxRegistry>` pre-loaded with `VimwikiSyntax`. Re-parse is full per change; incremental parsing deferred post-v1 per spec. Capabilities declared: - `text_document_sync = FULL` - `document_symbol_provider = true` - `position_encoding = UTF-8` when the client opts into LSP 3.17+ encodings (SPEC §6.11). Otherwise we keep the legacy UTF-16 default and translate spans through a per-line UTF-16 code-unit walk that handles BMP and surrogate pairs alike. Handlers: `initialize` (negotiate encoding), `initialized` (log it), `shutdown`, `did_open` / `did_change` (full re-parse + `publishDiagnostics`), `did_close` (drop state, clear diagnostics), `document_symbol` (nested outline). Diagnostics: every `BlockNode::Error` (from §6.7's resilient parser) emits one `DiagnosticSeverity::ERROR` with `source = "nuwiki"`. The walker descends into Blockquotes and ListItems so errors nested inside those don't slip through. Document outline: nested `DocumentSymbol` tree built recursively from heading levels (`= H1 =` → root, `== H2 ==` → child of preceding H1, etc.). Titles flatten inline markers via `inline_to_text` so `*world*` shows up as `world` in the outline. Pure helpers (`to_lsp_position`, `to_lsp_range`, `ast_diagnostics`, `headings_to_symbols`) are `pub` so the integration tests can verify conversion correctness without spinning up the async server. End-to- end LSP message exercises will land in Phase 8 alongside navigation. Tests (11): UTF-8 passthrough, UTF-16 conversion for é + 🦀, single and multi-error diagnostic emission, blockquote descent, flat outline, nested outline by levels, inline-marker stripping in titles, empty- heading fallback, registry-dispatch parity. Dep pin: `idna_adapter` held at 1.2.0 — 1.2.2 needs Cargo's edition2024 feature (Rust 1.85), our MSRV is 1.83. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |