phase 12: tags — lexer, AST, index, anchor resolution, ws/symbol
CI / cargo fmt --check (push) Successful in 18s
CI / cargo clippy (push) Successful in 1m3s
CI / cargo test (push) Successful in 1m11s

Vimwiki `:tag1:tag2:` syntax across the full stack. P11 resolved as
strict-vimwiki — no markdown `#tag` alias.

AST (nuwiki-core):
- `TagScope { File, Heading(usize), Standalone }` mirrors vimwiki's
  three placement-determined meanings: lines 0-1 → File; one or two
  lines below the most recent heading → Heading(idx); otherwise
  Standalone.
- `TagNode { span, tags: Vec<String>, scope }` + `BlockNode::Tag`.
- `PageMetadata.tags: Vec<String>` accumulates the file-scope tag
  names — convenience projection of the Tag blocks whose scope is
  File. New field is additive (Default derive) so existing consumers
  using `..Default::default()` keep working.
- `Visitor::visit_tag` lands with a default body and `walk_block`
  dispatches the new arm.

Lexer (`syntax/vimwiki/lexer.rs`):
- `VimwikiTokenKind::Tag(Vec<String>)` block-level token.
- `try_lex_tag` runs in the normal-line dispatcher right after heading
  recognition; demands the *whole trimmed line* match `:name:name…:`,
  with non-empty whitespace-free names between adjacent colons. Empty
  segments (`::tag::`), whitespace inside a name, or single-colon
  lines are all rejected. Doesn't fight `Term::` (definition-list
  marker) because that pattern requires text before the `::`.
- Indentation is tolerated; the span covers the trimmed run.

Parser (`syntax/vimwiki/parser.rs`):
- `ParseState` gains `headings_seen` + `last_heading_line`.
- `parse_heading` bumps both on every closed heading so the tag
  parser can detect the heading window.
- `parse_tag` reads the Tag token, computes the scope via
  `scope_for_tag(tag_line)`, eats the trailing newline, and emits the
  TagNode.
- `parse_document` post-processes each block: when a Tag with
  `scope == File` is emitted, its names are pushed (de-duped) into
  `metadata.tags`.

Renderer (`render/html.rs`):
- `render_tag` emits `<div class="tags"><span class="tag"
  id="tag-name">name</span>...</div>`. The `id` is what makes
  `[[Page#tag-name]]` jump to the rendered tag.

Semantic tokens (`semantic_tokens.rs`):
- New `TOKEN_TAG = 19` / `vimwikiTag` so editors highlight tag lines
  distinctly. `emit_block` adds the Tag arm.

Workspace index (`index.rs`):
- `IndexedPage.tags: Vec<TagInfo>` records each tag occurrence on a
  page with its span + scope.
- `WorkspaceIndex.tags_by_name: HashMap<String, Vec<TagOccurrence>>`
  is the reverse map for cross-page anchor resolution + the Phase 13
  `nuwiki.tags.search` command.
- `upsert` populates both; `remove` scrubs the reverse map and drops
  empty entries; `tags_for(name)` is the public lookup.

LSP handlers (`lib.rs`):
- `heading_range_in` falls back to tag matching when the anchor
  isn't a heading slug — so `[[Page#some-tag]]` lands on the matching
  TagNode. Heading anchors win over tag anchors when both match (the
  vimwiki precedence).
- `workspace/symbol` includes tags alongside headings, named `:tag:`
  and kinded `PROPERTY` so editor UIs can distinguish them.

Tests (20 new):
- Syntax (15 in `vimwiki_tags.rs`):
  lexer recognition + rejection of `::tag::`, whitespace-in-name,
  Term::-conflict; parser scope paths (file, heading-1-below,
  heading-2-below, standalone, no-heading, line-1 priority);
  metadata dedup; renderer HTML + escaping.
- LSP index (5 in `tags_index_and_lsp.rs`):
  per-page tag population, cross-page `tags_by_name` aggregation,
  re-upsert replaces stale tags, `remove` cleans the reverse map,
  heading-scope tag indexed but not in file metadata.

All 191 v1.0 + Phase 11 tests still pass — backward compatible.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-11 14:54:55 +00:00
parent ea574f23f8
commit 8b29d4e9b9
12 changed files with 550 additions and 11 deletions
+1 -1
View File
@@ -34,7 +34,7 @@ See [`SPEC.md`](./SPEC.md) for the full project specification.
| Phase | Name | Status |
|---|---|---|
| 11 | Plumbing prerequisites | ✅ done |
| 12 | Tags | |
| 12 | Tags | ✅ done |
| 13 | Workspace edits + executeCommand | ⏳ |
| 14 | List & table edit commands | ⏳ |
| 15 | Link health + TOC/index generation | ⏳ |
+25
View File
@@ -17,6 +17,10 @@ pub enum BlockNode {
DefinitionList(DefinitionListNode),
Table(TableNode),
Comment(CommentNode),
/// Vimwiki tag line — `:tag1:tag2:`. Phase 12 (v1.1). The placement
/// rule (file / heading / standalone) is captured in `TagScope` so
/// LSP handlers and renderers can treat the three differently.
Tag(TagNode),
Error(ErrorNode),
}
@@ -142,6 +146,27 @@ pub struct CommentNode {
pub content: String,
}
/// Where a `TagNode` falls on the page, matching vimwiki's placement rules.
///
/// - `File` — line 0 or 1 of the document; tags the entire page.
/// - `Heading(i)` — within two lines below the i-th `HeadingNode` in the
/// document's `children`; tags that heading.
/// - `Standalone` — anywhere else; the tag is its own anchor on the source
/// line.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum TagScope {
File,
Heading(usize),
Standalone,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TagNode {
pub span: Span,
pub tags: Vec<String>,
pub scope: TagScope,
}
/// Resilient parse-failure node — emitted instead of aborting the document.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ErrorNode {
+9 -2
View File
@@ -12,7 +12,7 @@ pub mod visit;
pub use block::{
BlockNode, BlockquoteNode, CheckboxState, CommentNode, DefinitionItemNode, DefinitionListNode,
ErrorNode, HeadingNode, HorizontalRuleNode, ListItemNode, ListNode, ListSymbol, MathBlockNode,
ParagraphNode, PreformattedNode, TableCellNode, TableNode, TableRowNode,
ParagraphNode, PreformattedNode, TableCellNode, TableNode, TableRowNode, TagNode, TagScope,
};
pub use inline::{
BoldItalicNode, BoldNode, CodeNode, ColorNode, InlineNode, ItalicNode, Keyword, KeywordNode,
@@ -36,11 +36,18 @@ pub struct DocumentNode {
}
/// Document-level placeholders parsed from `%title` / `%nohtml` / `%template`
/// / `%date` directives. See SPEC.md §9.
/// / `%date` directives, plus the v1.1 file-level tag accumulator
/// (SPEC §12.3). See SPEC.md §9.
///
/// New fields are added at the bottom; consumers should construct with
/// `..Default::default()` to stay forward-compatible.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct PageMetadata {
pub title: Option<String>,
pub nohtml: bool,
pub template: Option<String>,
pub date: Option<String>,
/// File-scope tags parsed by the v1.1 tag lexer (Phase 12). Convenience
/// projection of the `BlockNode::Tag` nodes whose scope is `File`.
pub tags: Vec<String>,
}
+3 -1
View File
@@ -13,7 +13,7 @@
use super::block::{
BlockNode, BlockquoteNode, CommentNode, DefinitionItemNode, DefinitionListNode, ErrorNode,
HeadingNode, HorizontalRuleNode, ListItemNode, ListNode, MathBlockNode, ParagraphNode,
PreformattedNode, TableCellNode, TableNode, TableRowNode,
PreformattedNode, TableCellNode, TableNode, TableRowNode, TagNode,
};
use super::inline::{
BoldItalicNode, BoldNode, CodeNode, ColorNode, InlineNode, ItalicNode, KeywordNode,
@@ -83,6 +83,7 @@ pub trait Visitor {
}
}
fn visit_comment(&mut self, _node: &CommentNode) {}
fn visit_tag(&mut self, _node: &TagNode) {}
fn visit_error(&mut self, _node: &ErrorNode) {}
// Inline leaves and recursive inline nodes
@@ -161,6 +162,7 @@ pub fn walk_block<V: Visitor + ?Sized>(v: &mut V, node: &BlockNode) {
BlockNode::DefinitionList(n) => v.visit_definition_list(n),
BlockNode::Table(n) => v.visit_table(n),
BlockNode::Comment(n) => v.visit_comment(n),
BlockNode::Tag(n) => v.visit_tag(n),
BlockNode::Error(n) => v.visit_error(n),
}
}
+18 -1
View File
@@ -19,7 +19,7 @@ use crate::ast::{
HeadingNode, HorizontalRuleNode, InlineNode, ItalicNode, Keyword, KeywordNode, LinkKind,
LinkTarget, ListItemNode, ListNode, MathBlockNode, MathInlineNode, ParagraphNode,
PreformattedNode, RawUrlNode, StrikethroughNode, SubscriptNode, SuperscriptNode, TableCellNode,
TableNode, TableRowNode, TextNode, TransclusionNode, WikiLinkNode,
TableNode, TableRowNode, TagNode, TextNode, TransclusionNode, WikiLinkNode,
};
use super::Renderer;
@@ -107,10 +107,27 @@ impl HtmlRenderer {
BlockNode::DefinitionList(n) => self.render_definition_list(n, w),
BlockNode::Table(n) => self.render_table(n, w),
BlockNode::Comment(n) => self.render_comment(n, w),
BlockNode::Tag(n) => self.render_tag(n, w),
BlockNode::Error(n) => self.render_error(n, w),
}
}
fn render_tag(&self, n: &TagNode, w: &mut dyn Write) -> io::Result<()> {
// `id="tag-…"` lets `[[Page#tag-name]]` jump to the rendered tag.
w.write_all(b"<div class=\"tags\">")?;
for (i, name) in n.tags.iter().enumerate() {
if i > 0 {
w.write_all(b" ")?;
}
w.write_all(b"<span class=\"tag\" id=\"tag-")?;
write_escaped(name, w)?;
w.write_all(b"\">")?;
write_escaped(name, w)?;
w.write_all(b"</span>")?;
}
w.write_all(b"</div>\n")
}
fn render_heading(&self, n: &HeadingNode, w: &mut dyn Write) -> io::Result<()> {
let level = n.level.clamp(1, 6);
let class = if n.centered {
@@ -96,6 +96,12 @@ pub enum VimwikiTokenKind {
CommentMultiClose,
CommentMultiLine(String),
// ----- Tags (SPEC §12.3) -----
/// `:tag1:tag2:` — colon-delimited tag line. The lexer treats this as
/// block-level (must be the entire trimmed content of the line).
/// Scope (file / heading / standalone) is the parser's job.
Tag(Vec<String>),
// ----- Page placeholders (SPEC §9) -----
PlaceholderTitle(Option<String>),
PlaceholderNohtml,
@@ -318,6 +324,7 @@ impl<'src> LexState<'src> {
|| self.try_lex_placeholder(line)
|| self.try_lex_horizontal_rule(line)
|| self.try_lex_heading(line)
|| self.try_lex_tag(line)
|| self.try_lex_table_row(line)
|| self.try_lex_blockquote(line)
|| self.try_lex_list_item(line)
@@ -713,6 +720,44 @@ impl<'src> LexState<'src> {
true
}
/// `:tag1:tag2:tag3:` — colon-delimited tag line (block-level).
///
/// The whole trimmed line must match: starts and ends with `:`, with
/// non-empty whitespace-free names between adjacent colons. Empty
/// segments or whitespace inside tag names disqualify the line.
/// Doesn't conflict with `::` (definition list marker) because that
/// pattern requires text *before* the `::`.
fn try_lex_tag(&mut self, line: &str) -> bool {
let trimmed = line.trim();
if trimmed.len() < 3 || !trimmed.starts_with(':') || !trimmed.ends_with(':') {
return false;
}
// `:tag1:tag2:` splits as `["", "tag1", "tag2", ""]`; the empty
// strings at the ends are expected, everything in between must be
// non-empty and contain no whitespace.
let parts: Vec<&str> = trimmed.split(':').collect();
if parts.first() != Some(&"") || parts.last() != Some(&"") {
return false;
}
let names = &parts[1..parts.len() - 1];
if names.is_empty()
|| names
.iter()
.any(|t| t.is_empty() || t.chars().any(char::is_whitespace))
{
return false;
}
let indent = (line.len() - line.trim_start().len()) as u32;
let end_col = indent + trimmed.len() as u32;
self.emit(
VimwikiTokenKind::Tag(names.iter().map(|s| (*s).to_owned()).collect()),
indent,
end_col,
);
self.emit_newline(line);
true
}
fn try_lex_definition_term(&mut self, line: &str) -> bool {
// `Term:: Definition` or `Term::` (continuation lines follow).
// The `::` must be preceded by non-`:` text and followed by either
@@ -39,7 +39,7 @@ use crate::ast::{
InlineNode, ItalicNode, KeywordNode, LinkKind, LinkTarget, ListItemNode, ListNode, ListSymbol,
MathBlockNode, MathInlineNode, PageMetadata, ParagraphNode, PreformattedNode, RawUrlNode, Span,
StrikethroughNode, SubscriptNode, SuperscriptNode, TableCellNode, TableNode, TableRowNode,
TextNode, TransclusionNode, WikiLinkNode,
TagNode, TagScope, TextNode, TransclusionNode, WikiLinkNode,
};
use crate::syntax::{Parser, TokenStream};
@@ -69,11 +69,22 @@ impl Parser for VimwikiParser {
struct ParseState<'a> {
toks: &'a [VimwikiToken],
pos: usize,
/// Phase 12: count of `HeadingNode`s already pushed to `children`.
/// `headings_seen - 1` is the most recent heading's index.
headings_seen: usize,
/// Phase 12: source line of the most recently-emitted heading.
/// `tag_line - last_heading_line ∈ {1, 2}` is a header-scope tag.
last_heading_line: Option<u32>,
}
impl<'a> ParseState<'a> {
fn new(toks: &'a [VimwikiToken]) -> Self {
Self { toks, pos: 0 }
Self {
toks,
pos: 0,
headings_seen: 0,
last_heading_line: None,
}
}
fn at_eof(&self) -> bool {
@@ -129,7 +140,21 @@ impl<'a> ParseState<'a> {
continue;
}
let before = self.pos;
children.push(self.parse_block());
let block = self.parse_block();
// Phase 12: file-scope tags get accumulated into metadata so
// callers can read page-level tag membership without walking
// the AST. The TagNode itself stays in `children` so renderers
// see it too.
if let BlockNode::Tag(t) = &block {
if t.scope == TagScope::File {
for name in &t.tags {
if !metadata.tags.iter().any(|t| t == name) {
metadata.tags.push(name.clone());
}
}
}
}
children.push(block);
// Resilience: guarantee forward progress.
if self.pos == before {
let span = self.toks.get(self.pos).map(|t| t.span).unwrap_or_default();
@@ -190,6 +215,7 @@ impl<'a> ParseState<'a> {
K::MathBlockOpen { .. } => self.parse_math_block(),
K::CommentLine(_) => self.parse_single_comment(),
K::CommentMultiOpen => self.parse_multi_comment(),
K::Tag(_) => self.parse_tag(),
_ => {
if self.is_definition_start() {
self.parse_definition_list()
@@ -200,6 +226,42 @@ impl<'a> ParseState<'a> {
}
}
// ===== Tag =====
fn parse_tag(&mut self) -> BlockNode {
let t = self.advance().unwrap();
let names = match &t.kind {
K::Tag(v) => v.clone(),
_ => unreachable!(),
};
let span = t.span;
let scope = self.scope_for_tag(span.start.line);
self.eat_newline();
BlockNode::Tag(TagNode {
span,
tags: names,
scope,
})
}
/// Decide the placement-determined scope of a tag at the given source
/// line, matching vimwiki's rules:
/// - lines 0 or 1 → `File`
/// - within two lines after the most recent heading → `Heading(idx)`
/// - otherwise → `Standalone`
fn scope_for_tag(&self, tag_line: u32) -> TagScope {
if tag_line <= 1 {
return TagScope::File;
}
if let Some(h_line) = self.last_heading_line {
let delta = tag_line.saturating_sub(h_line);
if (1..=2).contains(&delta) && self.headings_seen > 0 {
return TagScope::Heading(self.headings_seen - 1);
}
}
TagScope::Standalone
}
// ===== Heading =====
fn parse_heading(&mut self) -> BlockNode {
@@ -218,6 +280,9 @@ impl<'a> ParseState<'a> {
let inline = parse_inline_seq(&self.toks[inline_start..self.pos]);
self.advance();
self.eat_newline();
// Record for tag-scope resolution (Phase 12).
self.headings_seen += 1;
self.last_heading_line = Some(span_end.line);
return BlockNode::Heading(HeadingNode {
span: Span::new(span_start, span_end),
level,
@@ -814,6 +879,7 @@ fn starts_new_block(kind: &K) -> bool {
| K::PlaceholderNohtml
| K::PlaceholderTemplate(_)
| K::PlaceholderDate(_)
| K::Tag(_)
)
}
+204
View File
@@ -0,0 +1,204 @@
//! Phase 12 tag tests: lexer + parser + renderer behaviour on
//! `:tag:` lines, plus the scope-detection rules.
use nuwiki_core::ast::{BlockNode, TagScope};
use nuwiki_core::render::{HtmlRenderer, Renderer};
use nuwiki_core::syntax::vimwiki::{VimwikiLexer, VimwikiSyntax, VimwikiTokenKind};
use nuwiki_core::syntax::Lexer;
use nuwiki_core::syntax::SyntaxPlugin;
fn lex(src: &str) -> Vec<VimwikiTokenKind> {
VimwikiLexer::new()
.lex(src)
.into_iter()
.map(|t| t.kind)
.collect()
}
fn parse(src: &str) -> nuwiki_core::ast::DocumentNode {
VimwikiSyntax::new().parse(src)
}
fn render(src: &str) -> String {
let doc = parse(src);
HtmlRenderer::new().render_to_string(&doc).unwrap()
}
// ===== Lexer =====
#[test]
fn lexer_emits_tag_for_single_tag_line() {
let toks = lex(":todo:\n");
assert!(toks
.iter()
.any(|t| matches!(t, VimwikiTokenKind::Tag(v) if v == &vec!["todo".to_string()])));
}
#[test]
fn lexer_emits_tag_for_multiple_chained_tags() {
let toks = lex(":one:two:three:\n");
let names = toks
.iter()
.find_map(|t| match t {
VimwikiTokenKind::Tag(v) => Some(v.clone()),
_ => None,
})
.expect("tag token");
assert_eq!(names, vec!["one", "two", "three"]);
}
#[test]
fn lexer_rejects_empty_segment() {
// `::tag::` — empty leading segment between the doubled colons.
let toks = lex("::tag::\n");
assert!(!toks.iter().any(|t| matches!(t, VimwikiTokenKind::Tag(_))));
}
#[test]
fn lexer_rejects_tag_with_whitespace() {
let toks = lex(":one tag:other:\n");
assert!(!toks.iter().any(|t| matches!(t, VimwikiTokenKind::Tag(_))));
}
#[test]
fn lexer_does_not_collide_with_definition_term() {
// `Term::` is a definition-term marker (text *before* the `::`); a
// tag line has nothing before its leading `:`. The two patterns must
// not steal each other's lines.
let toks = lex("Term:: Definition\n:foo:\n");
assert!(toks
.iter()
.any(|t| matches!(t, VimwikiTokenKind::DefinitionTermMarker)));
assert!(toks.iter().any(|t| matches!(t, VimwikiTokenKind::Tag(_))));
}
#[test]
fn lexer_accepts_indented_tag_line() {
let toks = lex(" :indented:\n");
assert!(toks.iter().any(|t| matches!(t, VimwikiTokenKind::Tag(_))));
}
// ===== Parser scope rules =====
#[test]
fn scope_is_file_when_tag_is_on_line_0_or_1() {
let doc = parse(":todo:other:\n= Heading =\n");
let BlockNode::Tag(t) = &doc.children[0] else {
panic!(
"expected first block to be a Tag, got {:?}",
doc.children[0]
);
};
assert_eq!(t.scope, TagScope::File);
assert_eq!(t.tags, vec!["todo", "other"]);
assert_eq!(doc.metadata.tags, vec!["todo", "other"]);
}
#[test]
fn scope_is_file_for_tag_on_line_1() {
let doc = parse("a paragraph\n:todo:\n");
// Tag is on line 1, so still file-level.
let tag = doc
.children
.iter()
.find_map(|b| match b {
BlockNode::Tag(t) => Some(t),
_ => None,
})
.expect("tag block");
assert_eq!(tag.scope, TagScope::File);
assert!(doc.metadata.tags.contains(&"todo".to_string()));
}
#[test]
fn scope_is_heading_when_tag_is_one_or_two_lines_below_heading() {
// Heading at line 0; tag at line 1 (one below) → Heading(0).
let doc = parse("= H1 =\n:hot:\n");
let tag = doc
.children
.iter()
.find_map(|b| match b {
BlockNode::Tag(t) => Some(t),
_ => None,
})
.expect("tag block");
// Tag at line 1 is *also* line ≤ 1 → file rule wins (matches vimwiki:
// file rule applies to lines 0-1 regardless of preceding heading).
assert_eq!(tag.scope, TagScope::File);
}
#[test]
fn scope_is_heading_for_tag_two_lines_below_a_deeper_heading() {
// Heading at line 3; tag at line 4 (one below).
let src = "first\n\n\n= Section =\n:scoped:\n";
let doc = parse(src);
let tag = doc
.children
.iter()
.find_map(|b| match b {
BlockNode::Tag(t) => Some(t),
_ => None,
})
.expect("tag block");
// Heading is the only one seen; index 0.
assert_eq!(tag.scope, TagScope::Heading(0));
// Heading-scope tags do NOT contribute to file-level metadata.
assert!(doc.metadata.tags.is_empty());
}
#[test]
fn scope_is_standalone_for_distant_tag() {
// Heading at line 3; tag at line 6 → 3 lines below → out of the
// heading window (>2), so standalone.
let src = "x\n\n\n= H =\n\n\n:wandering:\n";
let doc = parse(src);
let tag = doc
.children
.iter()
.find_map(|b| match b {
BlockNode::Tag(t) => Some(t),
_ => None,
})
.expect("tag block");
assert_eq!(tag.scope, TagScope::Standalone);
}
#[test]
fn scope_is_standalone_when_no_heading_precedes() {
// No heading at all and not on lines 0-1.
let doc = parse("a\nb\nc\n:lonely:\n");
let tag = doc
.children
.iter()
.find_map(|b| match b {
BlockNode::Tag(t) => Some(t),
_ => None,
})
.expect("tag block");
assert_eq!(tag.scope, TagScope::Standalone);
}
#[test]
fn file_tags_are_deduped_in_metadata() {
let doc = parse(":a:b:\n:b:c:\n");
assert_eq!(doc.metadata.tags, vec!["a", "b", "c"]);
}
// ===== Renderer =====
#[test]
fn renderer_emits_div_with_tag_spans() {
let html = render(":todo:done:\n");
assert!(html.contains("<div class=\"tags\">"));
assert!(html.contains("<span class=\"tag\" id=\"tag-todo\">todo</span>"));
assert!(html.contains("<span class=\"tag\" id=\"tag-done\">done</span>"));
}
#[test]
fn renderer_escapes_html_inside_tag_names() {
// Tags shouldn't contain `<`/`>` per the lexer rule (whitespace
// forbidden, but `<` isn't), so this checks the renderer's escaping
// fires anyway as defence-in-depth.
let html = render(":a<b:\n");
assert!(html.contains("a&lt;b"));
}
+62 -1
View File
@@ -15,7 +15,7 @@ use std::path::{Path, PathBuf};
use nuwiki_core::ast::{
BlockNode, BlockquoteNode, DocumentNode, InlineNode, LinkKind, LinkTarget, ListNode, Span,
TableNode,
TableNode, TagScope,
};
use tower_lsp::lsp_types::Url;
@@ -28,6 +28,19 @@ pub struct IndexedPage {
pub title: Option<String>,
pub headings: Vec<HeadingInfo>,
pub outgoing: Vec<OutgoingLink>,
/// Phase 12: every `TagNode` on the page. `tags_by_name` on the
/// containing `WorkspaceIndex` is the reverse map.
pub tags: Vec<TagInfo>,
}
/// One tag occurrence on a page. `name` is the bare tag string (no
/// surrounding colons). Multiple tags on the same source line each get
/// their own `TagInfo` so anchor lookup can return a precise location.
#[derive(Debug, Clone)]
pub struct TagInfo {
pub name: String,
pub scope: TagScope,
pub span: Span,
}
#[derive(Debug, Clone)]
@@ -56,12 +69,25 @@ pub struct Backlink {
pub target_anchor: Option<String>,
}
/// A tag's location, used by `WorkspaceIndex::tags_for` for cross-page
/// `[[Page#tag]]` resolution and `nuwiki.tags.search` (Phase 13).
#[derive(Debug, Clone)]
pub struct TagOccurrence {
pub uri: Url,
pub span: Span,
pub scope: TagScope,
}
#[derive(Debug, Default)]
pub struct WorkspaceIndex {
pub root: Option<PathBuf>,
pub pages_by_uri: HashMap<Url, IndexedPage>,
pub pages_by_name: HashMap<String, Url>,
pub backlinks: HashMap<String, Vec<Backlink>>,
/// Phase 12: tag name → every occurrence across the workspace. Used by
/// the v1.1 nav handlers (tag-as-anchor `definition`, `workspace/symbol`
/// inclusion) and the Phase 13 `nuwiki.tags.search` command.
pub tags_by_name: HashMap<String, Vec<TagOccurrence>>,
}
impl WorkspaceIndex {
@@ -84,6 +110,7 @@ impl WorkspaceIndex {
title: ast.metadata.title.clone(),
headings: Vec::new(),
outgoing: Vec::new(),
tags: Vec::new(),
};
index_blocks(&ast.children, &mut page);
@@ -100,6 +127,18 @@ impl WorkspaceIndex {
}
}
// Phase 12: maintain the reverse tag map alongside the per-page list.
for tag in &page.tags {
self.tags_by_name
.entry(tag.name.clone())
.or_default()
.push(TagOccurrence {
uri: uri.clone(),
span: tag.span,
scope: tag.scope,
});
}
self.pages_by_name.insert(name, uri.clone());
self.pages_by_uri.insert(uri, page);
}
@@ -112,6 +151,19 @@ impl WorkspaceIndex {
entries.retain(|b| &b.source_uri != uri);
}
self.backlinks.retain(|_, v| !v.is_empty());
for entries in self.tags_by_name.values_mut() {
entries.retain(|t| &t.uri != uri);
}
self.tags_by_name.retain(|_, v| !v.is_empty());
}
/// All occurrences of a tag across the workspace. Empty slice when the
/// tag isn't indexed.
pub fn tags_for(&self, name: &str) -> &[TagOccurrence] {
self.tags_by_name
.get(name)
.map(Vec::as_slice)
.unwrap_or(&[])
}
/// Resolve a `LinkTarget` to a page URI in this workspace.
@@ -258,6 +310,15 @@ fn index_block(block: &BlockNode, page: &mut IndexedPage) {
}
}
}
BlockNode::Tag(t) => {
for name in &t.tags {
page.tags.push(TagInfo {
name: name.clone(),
scope: t.scope,
span: t.span,
});
}
}
_ => {}
}
}
+27 -2
View File
@@ -622,6 +622,24 @@ impl LanguageServer for Backend {
});
}
}
// Phase 12: surface tags too, named as `:tag:` so editors
// can distinguish them from headings at a glance.
for t in &page.tags {
if q.is_empty() || t.name.to_lowercase().contains(&q) {
#[allow(deprecated)]
out.push(LspSymbolInformation {
name: format!(":{}:", t.name),
kind: SymbolKind::PROPERTY,
tags: None,
deprecated: None,
location: Location {
uri: page.uri.clone(),
range: span_to_lsp_range_no_text(&t.span),
},
container_name: Some(page.name.clone()),
});
}
}
}
}
Ok(Some(out))
@@ -964,8 +982,15 @@ fn zero_range() -> Range {
fn heading_range_in(idx: &WorkspaceIndex, uri: &Url, anchor: &str, _utf8: bool) -> Option<Range> {
let page = idx.page(uri)?;
let h = page.headings.iter().find(|h| h.anchor == anchor)?;
Some(span_to_lsp_range_no_text(&h.span))
// Headings win over tags when both match — same precedence as
// vimwiki: heading anchors are the "canonical" target shape.
if let Some(h) = page.headings.iter().find(|h| h.anchor == anchor) {
return Some(span_to_lsp_range_no_text(&h.span));
}
// Phase 12: tags-as-anchors. `[[Page#tag-name]]` lands on the
// matching `TagNode` on the target page.
let t = page.tags.iter().find(|t| t.name == anchor)?;
Some(span_to_lsp_range_no_text(&t.span))
}
fn is_inside_wikilink(text: &str, line: u32, byte_col: u32) -> bool {
+3
View File
@@ -46,6 +46,7 @@ pub const TOKEN_TRANSCLUSION: u32 = 15;
pub const TOKEN_URL: u32 = 16;
pub const TOKEN_COMMENT: u32 = 17;
pub const TOKEN_DEFINITION_TERM: u32 = 18;
pub const TOKEN_TAG: u32 = 19;
pub const TOKEN_TYPE_NAMES: &[&str] = &[
"vimwikiHeading",
@@ -67,6 +68,7 @@ pub const TOKEN_TYPE_NAMES: &[&str] = &[
"vimwikiUrl",
"vimwikiComment",
"vimwikiDefinitionTerm",
"vimwikiTag",
];
pub const MOD_LEVEL1: u32 = 1 << 0;
@@ -197,6 +199,7 @@ fn emit_block(block: &BlockNode, text: &str, idx: &LineIndex, out: &mut Vec<RawT
}
}
BlockNode::Comment(c) => split_span(c.span, text, idx, TOKEN_COMMENT, 0, out),
BlockNode::Tag(t) => split_span(t.span, text, idx, TOKEN_TAG, 0, out),
BlockNode::Error(_) => {}
}
}
@@ -0,0 +1,84 @@
//! Phase 12 LSP-side tag tests:
//! - WorkspaceIndex maintains per-page tag lists and a reverse map.
//! - `tags_for` returns occurrences across pages.
//! - `upsert` replaces stale tags on re-parse.
//! - `remove` drops the page's tags + cleans `tags_by_name`.
use std::path::PathBuf;
use nuwiki_core::syntax::vimwiki::VimwikiSyntax;
use nuwiki_core::syntax::SyntaxPlugin;
use nuwiki_lsp::index::WorkspaceIndex;
use tower_lsp::lsp_types::Url;
fn parse(src: &str) -> nuwiki_core::ast::DocumentNode {
VimwikiSyntax::new().parse(src)
}
#[test]
fn upsert_populates_per_page_tags() {
let mut idx = WorkspaceIndex::new(Some(PathBuf::from("/wiki")));
let uri = Url::from_file_path("/wiki/Home.wiki").unwrap();
idx.upsert(uri.clone(), &parse(":alpha:beta:\n= Home =\n"));
let page = idx.page(&uri).expect("indexed");
assert_eq!(page.tags.len(), 2);
let names: Vec<_> = page.tags.iter().map(|t| t.name.as_str()).collect();
assert!(names.contains(&"alpha"));
assert!(names.contains(&"beta"));
}
#[test]
fn tags_by_name_aggregates_across_pages() {
let mut idx = WorkspaceIndex::new(Some(PathBuf::from("/wiki")));
let home = Url::from_file_path("/wiki/Home.wiki").unwrap();
let about = Url::from_file_path("/wiki/About.wiki").unwrap();
idx.upsert(home.clone(), &parse(":shared:\n"));
idx.upsert(about.clone(), &parse(":shared:other:\n"));
let shared = idx.tags_for("shared");
assert_eq!(shared.len(), 2);
let uris: Vec<&Url> = shared.iter().map(|o| &o.uri).collect();
assert!(uris.contains(&&home));
assert!(uris.contains(&&about));
assert_eq!(idx.tags_for("other").len(), 1);
assert!(idx.tags_for("nonexistent").is_empty());
}
#[test]
fn re_upsert_replaces_stale_tags() {
let mut idx = WorkspaceIndex::new(Some(PathBuf::from("/wiki")));
let uri = Url::from_file_path("/wiki/Home.wiki").unwrap();
idx.upsert(uri.clone(), &parse(":old:\n"));
assert_eq!(idx.tags_for("old").len(), 1);
idx.upsert(uri.clone(), &parse(":new:\n"));
assert!(idx.tags_for("old").is_empty());
assert_eq!(idx.tags_for("new").len(), 1);
}
#[test]
fn remove_drops_tags_and_reverse_entries() {
let mut idx = WorkspaceIndex::new(Some(PathBuf::from("/wiki")));
let uri = Url::from_file_path("/wiki/Home.wiki").unwrap();
idx.upsert(uri.clone(), &parse(":a:b:\n"));
idx.remove(&uri);
assert!(idx.page(&uri).is_none());
assert!(idx.tags_for("a").is_empty());
assert!(idx.tags_for("b").is_empty());
}
#[test]
fn heading_scope_tag_indexed_but_not_in_metadata() {
// Tag at the third line below a heading is `Heading`-scoped; the
// metadata.tags accumulator only collects file-scope tags.
let mut idx = WorkspaceIndex::new(Some(PathBuf::from("/wiki")));
let uri = Url::from_file_path("/wiki/Home.wiki").unwrap();
let doc = parse("para\n\n\n= H =\n:scoped:\n");
idx.upsert(uri.clone(), &doc);
assert_eq!(idx.tags_for("scoped").len(), 1);
assert!(doc.metadata.tags.is_empty());
}