phase 12: tags — lexer, AST, index, anchor resolution, ws/symbol
Vimwiki `:tag1:tag2:` syntax across the full stack. P11 resolved as
strict-vimwiki — no markdown `#tag` alias.
AST (nuwiki-core):
- `TagScope { File, Heading(usize), Standalone }` mirrors vimwiki's
three placement-determined meanings: lines 0-1 → File; one or two
lines below the most recent heading → Heading(idx); otherwise
Standalone.
- `TagNode { span, tags: Vec<String>, scope }` + `BlockNode::Tag`.
- `PageMetadata.tags: Vec<String>` accumulates the file-scope tag
names — convenience projection of the Tag blocks whose scope is
File. New field is additive (Default derive) so existing consumers
using `..Default::default()` keep working.
- `Visitor::visit_tag` lands with a default body and `walk_block`
dispatches the new arm.
Lexer (`syntax/vimwiki/lexer.rs`):
- `VimwikiTokenKind::Tag(Vec<String>)` block-level token.
- `try_lex_tag` runs in the normal-line dispatcher right after heading
recognition; demands the *whole trimmed line* match `:name:name…:`,
with non-empty whitespace-free names between adjacent colons. Empty
segments (`::tag::`), whitespace inside a name, or single-colon
lines are all rejected. Doesn't fight `Term::` (definition-list
marker) because that pattern requires text before the `::`.
- Indentation is tolerated; the span covers the trimmed run.
Parser (`syntax/vimwiki/parser.rs`):
- `ParseState` gains `headings_seen` + `last_heading_line`.
- `parse_heading` bumps both on every closed heading so the tag
parser can detect the heading window.
- `parse_tag` reads the Tag token, computes the scope via
`scope_for_tag(tag_line)`, eats the trailing newline, and emits the
TagNode.
- `parse_document` post-processes each block: when a Tag with
`scope == File` is emitted, its names are pushed (de-duped) into
`metadata.tags`.
Renderer (`render/html.rs`):
- `render_tag` emits `<div class="tags"><span class="tag"
id="tag-name">name</span>...</div>`. The `id` is what makes
`[[Page#tag-name]]` jump to the rendered tag.
Semantic tokens (`semantic_tokens.rs`):
- New `TOKEN_TAG = 19` / `vimwikiTag` so editors highlight tag lines
distinctly. `emit_block` adds the Tag arm.
Workspace index (`index.rs`):
- `IndexedPage.tags: Vec<TagInfo>` records each tag occurrence on a
page with its span + scope.
- `WorkspaceIndex.tags_by_name: HashMap<String, Vec<TagOccurrence>>`
is the reverse map for cross-page anchor resolution + the Phase 13
`nuwiki.tags.search` command.
- `upsert` populates both; `remove` scrubs the reverse map and drops
empty entries; `tags_for(name)` is the public lookup.
LSP handlers (`lib.rs`):
- `heading_range_in` falls back to tag matching when the anchor
isn't a heading slug — so `[[Page#some-tag]]` lands on the matching
TagNode. Heading anchors win over tag anchors when both match (the
vimwiki precedence).
- `workspace/symbol` includes tags alongside headings, named `:tag:`
and kinded `PROPERTY` so editor UIs can distinguish them.
Tests (20 new):
- Syntax (15 in `vimwiki_tags.rs`):
lexer recognition + rejection of `::tag::`, whitespace-in-name,
Term::-conflict; parser scope paths (file, heading-1-below,
heading-2-below, standalone, no-heading, line-1 priority);
metadata dedup; renderer HTML + escaping.
- LSP index (5 in `tags_index_and_lsp.rs`):
per-page tag population, cross-page `tags_by_name` aggregation,
re-upsert replaces stale tags, `remove` cleans the reverse map,
heading-scope tag indexed but not in file metadata.
All 191 v1.0 + Phase 11 tests still pass — backward compatible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -17,6 +17,10 @@ pub enum BlockNode {
|
||||
DefinitionList(DefinitionListNode),
|
||||
Table(TableNode),
|
||||
Comment(CommentNode),
|
||||
/// Vimwiki tag line — `:tag1:tag2:`. Phase 12 (v1.1). The placement
|
||||
/// rule (file / heading / standalone) is captured in `TagScope` so
|
||||
/// LSP handlers and renderers can treat the three differently.
|
||||
Tag(TagNode),
|
||||
Error(ErrorNode),
|
||||
}
|
||||
|
||||
@@ -142,6 +146,27 @@ pub struct CommentNode {
|
||||
pub content: String,
|
||||
}
|
||||
|
||||
/// Where a `TagNode` falls on the page, matching vimwiki's placement rules.
|
||||
///
|
||||
/// - `File` — line 0 or 1 of the document; tags the entire page.
|
||||
/// - `Heading(i)` — within two lines below the i-th `HeadingNode` in the
|
||||
/// document's `children`; tags that heading.
|
||||
/// - `Standalone` — anywhere else; the tag is its own anchor on the source
|
||||
/// line.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||
pub enum TagScope {
|
||||
File,
|
||||
Heading(usize),
|
||||
Standalone,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct TagNode {
|
||||
pub span: Span,
|
||||
pub tags: Vec<String>,
|
||||
pub scope: TagScope,
|
||||
}
|
||||
|
||||
/// Resilient parse-failure node — emitted instead of aborting the document.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ErrorNode {
|
||||
|
||||
@@ -12,7 +12,7 @@ pub mod visit;
|
||||
pub use block::{
|
||||
BlockNode, BlockquoteNode, CheckboxState, CommentNode, DefinitionItemNode, DefinitionListNode,
|
||||
ErrorNode, HeadingNode, HorizontalRuleNode, ListItemNode, ListNode, ListSymbol, MathBlockNode,
|
||||
ParagraphNode, PreformattedNode, TableCellNode, TableNode, TableRowNode,
|
||||
ParagraphNode, PreformattedNode, TableCellNode, TableNode, TableRowNode, TagNode, TagScope,
|
||||
};
|
||||
pub use inline::{
|
||||
BoldItalicNode, BoldNode, CodeNode, ColorNode, InlineNode, ItalicNode, Keyword, KeywordNode,
|
||||
@@ -36,11 +36,18 @@ pub struct DocumentNode {
|
||||
}
|
||||
|
||||
/// Document-level placeholders parsed from `%title` / `%nohtml` / `%template`
|
||||
/// / `%date` directives. See SPEC.md §9.
|
||||
/// / `%date` directives, plus the v1.1 file-level tag accumulator
|
||||
/// (SPEC §12.3). See SPEC.md §9.
|
||||
///
|
||||
/// New fields are added at the bottom; consumers should construct with
|
||||
/// `..Default::default()` to stay forward-compatible.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq)]
|
||||
pub struct PageMetadata {
|
||||
pub title: Option<String>,
|
||||
pub nohtml: bool,
|
||||
pub template: Option<String>,
|
||||
pub date: Option<String>,
|
||||
/// File-scope tags parsed by the v1.1 tag lexer (Phase 12). Convenience
|
||||
/// projection of the `BlockNode::Tag` nodes whose scope is `File`.
|
||||
pub tags: Vec<String>,
|
||||
}
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
use super::block::{
|
||||
BlockNode, BlockquoteNode, CommentNode, DefinitionItemNode, DefinitionListNode, ErrorNode,
|
||||
HeadingNode, HorizontalRuleNode, ListItemNode, ListNode, MathBlockNode, ParagraphNode,
|
||||
PreformattedNode, TableCellNode, TableNode, TableRowNode,
|
||||
PreformattedNode, TableCellNode, TableNode, TableRowNode, TagNode,
|
||||
};
|
||||
use super::inline::{
|
||||
BoldItalicNode, BoldNode, CodeNode, ColorNode, InlineNode, ItalicNode, KeywordNode,
|
||||
@@ -83,6 +83,7 @@ pub trait Visitor {
|
||||
}
|
||||
}
|
||||
fn visit_comment(&mut self, _node: &CommentNode) {}
|
||||
fn visit_tag(&mut self, _node: &TagNode) {}
|
||||
fn visit_error(&mut self, _node: &ErrorNode) {}
|
||||
|
||||
// Inline leaves and recursive inline nodes
|
||||
@@ -161,6 +162,7 @@ pub fn walk_block<V: Visitor + ?Sized>(v: &mut V, node: &BlockNode) {
|
||||
BlockNode::DefinitionList(n) => v.visit_definition_list(n),
|
||||
BlockNode::Table(n) => v.visit_table(n),
|
||||
BlockNode::Comment(n) => v.visit_comment(n),
|
||||
BlockNode::Tag(n) => v.visit_tag(n),
|
||||
BlockNode::Error(n) => v.visit_error(n),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,7 +19,7 @@ use crate::ast::{
|
||||
HeadingNode, HorizontalRuleNode, InlineNode, ItalicNode, Keyword, KeywordNode, LinkKind,
|
||||
LinkTarget, ListItemNode, ListNode, MathBlockNode, MathInlineNode, ParagraphNode,
|
||||
PreformattedNode, RawUrlNode, StrikethroughNode, SubscriptNode, SuperscriptNode, TableCellNode,
|
||||
TableNode, TableRowNode, TextNode, TransclusionNode, WikiLinkNode,
|
||||
TableNode, TableRowNode, TagNode, TextNode, TransclusionNode, WikiLinkNode,
|
||||
};
|
||||
|
||||
use super::Renderer;
|
||||
@@ -107,10 +107,27 @@ impl HtmlRenderer {
|
||||
BlockNode::DefinitionList(n) => self.render_definition_list(n, w),
|
||||
BlockNode::Table(n) => self.render_table(n, w),
|
||||
BlockNode::Comment(n) => self.render_comment(n, w),
|
||||
BlockNode::Tag(n) => self.render_tag(n, w),
|
||||
BlockNode::Error(n) => self.render_error(n, w),
|
||||
}
|
||||
}
|
||||
|
||||
fn render_tag(&self, n: &TagNode, w: &mut dyn Write) -> io::Result<()> {
|
||||
// `id="tag-…"` lets `[[Page#tag-name]]` jump to the rendered tag.
|
||||
w.write_all(b"<div class=\"tags\">")?;
|
||||
for (i, name) in n.tags.iter().enumerate() {
|
||||
if i > 0 {
|
||||
w.write_all(b" ")?;
|
||||
}
|
||||
w.write_all(b"<span class=\"tag\" id=\"tag-")?;
|
||||
write_escaped(name, w)?;
|
||||
w.write_all(b"\">")?;
|
||||
write_escaped(name, w)?;
|
||||
w.write_all(b"</span>")?;
|
||||
}
|
||||
w.write_all(b"</div>\n")
|
||||
}
|
||||
|
||||
fn render_heading(&self, n: &HeadingNode, w: &mut dyn Write) -> io::Result<()> {
|
||||
let level = n.level.clamp(1, 6);
|
||||
let class = if n.centered {
|
||||
|
||||
@@ -96,6 +96,12 @@ pub enum VimwikiTokenKind {
|
||||
CommentMultiClose,
|
||||
CommentMultiLine(String),
|
||||
|
||||
// ----- Tags (SPEC §12.3) -----
|
||||
/// `:tag1:tag2:` — colon-delimited tag line. The lexer treats this as
|
||||
/// block-level (must be the entire trimmed content of the line).
|
||||
/// Scope (file / heading / standalone) is the parser's job.
|
||||
Tag(Vec<String>),
|
||||
|
||||
// ----- Page placeholders (SPEC §9) -----
|
||||
PlaceholderTitle(Option<String>),
|
||||
PlaceholderNohtml,
|
||||
@@ -318,6 +324,7 @@ impl<'src> LexState<'src> {
|
||||
|| self.try_lex_placeholder(line)
|
||||
|| self.try_lex_horizontal_rule(line)
|
||||
|| self.try_lex_heading(line)
|
||||
|| self.try_lex_tag(line)
|
||||
|| self.try_lex_table_row(line)
|
||||
|| self.try_lex_blockquote(line)
|
||||
|| self.try_lex_list_item(line)
|
||||
@@ -713,6 +720,44 @@ impl<'src> LexState<'src> {
|
||||
true
|
||||
}
|
||||
|
||||
/// `:tag1:tag2:tag3:` — colon-delimited tag line (block-level).
|
||||
///
|
||||
/// The whole trimmed line must match: starts and ends with `:`, with
|
||||
/// non-empty whitespace-free names between adjacent colons. Empty
|
||||
/// segments or whitespace inside tag names disqualify the line.
|
||||
/// Doesn't conflict with `::` (definition list marker) because that
|
||||
/// pattern requires text *before* the `::`.
|
||||
fn try_lex_tag(&mut self, line: &str) -> bool {
|
||||
let trimmed = line.trim();
|
||||
if trimmed.len() < 3 || !trimmed.starts_with(':') || !trimmed.ends_with(':') {
|
||||
return false;
|
||||
}
|
||||
// `:tag1:tag2:` splits as `["", "tag1", "tag2", ""]`; the empty
|
||||
// strings at the ends are expected, everything in between must be
|
||||
// non-empty and contain no whitespace.
|
||||
let parts: Vec<&str> = trimmed.split(':').collect();
|
||||
if parts.first() != Some(&"") || parts.last() != Some(&"") {
|
||||
return false;
|
||||
}
|
||||
let names = &parts[1..parts.len() - 1];
|
||||
if names.is_empty()
|
||||
|| names
|
||||
.iter()
|
||||
.any(|t| t.is_empty() || t.chars().any(char::is_whitespace))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
let indent = (line.len() - line.trim_start().len()) as u32;
|
||||
let end_col = indent + trimmed.len() as u32;
|
||||
self.emit(
|
||||
VimwikiTokenKind::Tag(names.iter().map(|s| (*s).to_owned()).collect()),
|
||||
indent,
|
||||
end_col,
|
||||
);
|
||||
self.emit_newline(line);
|
||||
true
|
||||
}
|
||||
|
||||
fn try_lex_definition_term(&mut self, line: &str) -> bool {
|
||||
// `Term:: Definition` or `Term::` (continuation lines follow).
|
||||
// The `::` must be preceded by non-`:` text and followed by either
|
||||
|
||||
@@ -39,7 +39,7 @@ use crate::ast::{
|
||||
InlineNode, ItalicNode, KeywordNode, LinkKind, LinkTarget, ListItemNode, ListNode, ListSymbol,
|
||||
MathBlockNode, MathInlineNode, PageMetadata, ParagraphNode, PreformattedNode, RawUrlNode, Span,
|
||||
StrikethroughNode, SubscriptNode, SuperscriptNode, TableCellNode, TableNode, TableRowNode,
|
||||
TextNode, TransclusionNode, WikiLinkNode,
|
||||
TagNode, TagScope, TextNode, TransclusionNode, WikiLinkNode,
|
||||
};
|
||||
use crate::syntax::{Parser, TokenStream};
|
||||
|
||||
@@ -69,11 +69,22 @@ impl Parser for VimwikiParser {
|
||||
struct ParseState<'a> {
|
||||
toks: &'a [VimwikiToken],
|
||||
pos: usize,
|
||||
/// Phase 12: count of `HeadingNode`s already pushed to `children`.
|
||||
/// `headings_seen - 1` is the most recent heading's index.
|
||||
headings_seen: usize,
|
||||
/// Phase 12: source line of the most recently-emitted heading.
|
||||
/// `tag_line - last_heading_line ∈ {1, 2}` is a header-scope tag.
|
||||
last_heading_line: Option<u32>,
|
||||
}
|
||||
|
||||
impl<'a> ParseState<'a> {
|
||||
fn new(toks: &'a [VimwikiToken]) -> Self {
|
||||
Self { toks, pos: 0 }
|
||||
Self {
|
||||
toks,
|
||||
pos: 0,
|
||||
headings_seen: 0,
|
||||
last_heading_line: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn at_eof(&self) -> bool {
|
||||
@@ -129,7 +140,21 @@ impl<'a> ParseState<'a> {
|
||||
continue;
|
||||
}
|
||||
let before = self.pos;
|
||||
children.push(self.parse_block());
|
||||
let block = self.parse_block();
|
||||
// Phase 12: file-scope tags get accumulated into metadata so
|
||||
// callers can read page-level tag membership without walking
|
||||
// the AST. The TagNode itself stays in `children` so renderers
|
||||
// see it too.
|
||||
if let BlockNode::Tag(t) = &block {
|
||||
if t.scope == TagScope::File {
|
||||
for name in &t.tags {
|
||||
if !metadata.tags.iter().any(|t| t == name) {
|
||||
metadata.tags.push(name.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
children.push(block);
|
||||
// Resilience: guarantee forward progress.
|
||||
if self.pos == before {
|
||||
let span = self.toks.get(self.pos).map(|t| t.span).unwrap_or_default();
|
||||
@@ -190,6 +215,7 @@ impl<'a> ParseState<'a> {
|
||||
K::MathBlockOpen { .. } => self.parse_math_block(),
|
||||
K::CommentLine(_) => self.parse_single_comment(),
|
||||
K::CommentMultiOpen => self.parse_multi_comment(),
|
||||
K::Tag(_) => self.parse_tag(),
|
||||
_ => {
|
||||
if self.is_definition_start() {
|
||||
self.parse_definition_list()
|
||||
@@ -200,6 +226,42 @@ impl<'a> ParseState<'a> {
|
||||
}
|
||||
}
|
||||
|
||||
// ===== Tag =====
|
||||
|
||||
fn parse_tag(&mut self) -> BlockNode {
|
||||
let t = self.advance().unwrap();
|
||||
let names = match &t.kind {
|
||||
K::Tag(v) => v.clone(),
|
||||
_ => unreachable!(),
|
||||
};
|
||||
let span = t.span;
|
||||
let scope = self.scope_for_tag(span.start.line);
|
||||
self.eat_newline();
|
||||
BlockNode::Tag(TagNode {
|
||||
span,
|
||||
tags: names,
|
||||
scope,
|
||||
})
|
||||
}
|
||||
|
||||
/// Decide the placement-determined scope of a tag at the given source
|
||||
/// line, matching vimwiki's rules:
|
||||
/// - lines 0 or 1 → `File`
|
||||
/// - within two lines after the most recent heading → `Heading(idx)`
|
||||
/// - otherwise → `Standalone`
|
||||
fn scope_for_tag(&self, tag_line: u32) -> TagScope {
|
||||
if tag_line <= 1 {
|
||||
return TagScope::File;
|
||||
}
|
||||
if let Some(h_line) = self.last_heading_line {
|
||||
let delta = tag_line.saturating_sub(h_line);
|
||||
if (1..=2).contains(&delta) && self.headings_seen > 0 {
|
||||
return TagScope::Heading(self.headings_seen - 1);
|
||||
}
|
||||
}
|
||||
TagScope::Standalone
|
||||
}
|
||||
|
||||
// ===== Heading =====
|
||||
|
||||
fn parse_heading(&mut self) -> BlockNode {
|
||||
@@ -218,6 +280,9 @@ impl<'a> ParseState<'a> {
|
||||
let inline = parse_inline_seq(&self.toks[inline_start..self.pos]);
|
||||
self.advance();
|
||||
self.eat_newline();
|
||||
// Record for tag-scope resolution (Phase 12).
|
||||
self.headings_seen += 1;
|
||||
self.last_heading_line = Some(span_end.line);
|
||||
return BlockNode::Heading(HeadingNode {
|
||||
span: Span::new(span_start, span_end),
|
||||
level,
|
||||
@@ -814,6 +879,7 @@ fn starts_new_block(kind: &K) -> bool {
|
||||
| K::PlaceholderNohtml
|
||||
| K::PlaceholderTemplate(_)
|
||||
| K::PlaceholderDate(_)
|
||||
| K::Tag(_)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
//! Phase 12 tag tests: lexer + parser + renderer behaviour on
|
||||
//! `:tag:` lines, plus the scope-detection rules.
|
||||
|
||||
use nuwiki_core::ast::{BlockNode, TagScope};
|
||||
use nuwiki_core::render::{HtmlRenderer, Renderer};
|
||||
use nuwiki_core::syntax::vimwiki::{VimwikiLexer, VimwikiSyntax, VimwikiTokenKind};
|
||||
use nuwiki_core::syntax::Lexer;
|
||||
use nuwiki_core::syntax::SyntaxPlugin;
|
||||
|
||||
fn lex(src: &str) -> Vec<VimwikiTokenKind> {
|
||||
VimwikiLexer::new()
|
||||
.lex(src)
|
||||
.into_iter()
|
||||
.map(|t| t.kind)
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn parse(src: &str) -> nuwiki_core::ast::DocumentNode {
|
||||
VimwikiSyntax::new().parse(src)
|
||||
}
|
||||
|
||||
fn render(src: &str) -> String {
|
||||
let doc = parse(src);
|
||||
HtmlRenderer::new().render_to_string(&doc).unwrap()
|
||||
}
|
||||
|
||||
// ===== Lexer =====
|
||||
|
||||
#[test]
|
||||
fn lexer_emits_tag_for_single_tag_line() {
|
||||
let toks = lex(":todo:\n");
|
||||
assert!(toks
|
||||
.iter()
|
||||
.any(|t| matches!(t, VimwikiTokenKind::Tag(v) if v == &vec!["todo".to_string()])));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lexer_emits_tag_for_multiple_chained_tags() {
|
||||
let toks = lex(":one:two:three:\n");
|
||||
let names = toks
|
||||
.iter()
|
||||
.find_map(|t| match t {
|
||||
VimwikiTokenKind::Tag(v) => Some(v.clone()),
|
||||
_ => None,
|
||||
})
|
||||
.expect("tag token");
|
||||
assert_eq!(names, vec!["one", "two", "three"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lexer_rejects_empty_segment() {
|
||||
// `::tag::` — empty leading segment between the doubled colons.
|
||||
let toks = lex("::tag::\n");
|
||||
assert!(!toks.iter().any(|t| matches!(t, VimwikiTokenKind::Tag(_))));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lexer_rejects_tag_with_whitespace() {
|
||||
let toks = lex(":one tag:other:\n");
|
||||
assert!(!toks.iter().any(|t| matches!(t, VimwikiTokenKind::Tag(_))));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lexer_does_not_collide_with_definition_term() {
|
||||
// `Term::` is a definition-term marker (text *before* the `::`); a
|
||||
// tag line has nothing before its leading `:`. The two patterns must
|
||||
// not steal each other's lines.
|
||||
let toks = lex("Term:: Definition\n:foo:\n");
|
||||
assert!(toks
|
||||
.iter()
|
||||
.any(|t| matches!(t, VimwikiTokenKind::DefinitionTermMarker)));
|
||||
assert!(toks.iter().any(|t| matches!(t, VimwikiTokenKind::Tag(_))));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lexer_accepts_indented_tag_line() {
|
||||
let toks = lex(" :indented:\n");
|
||||
assert!(toks.iter().any(|t| matches!(t, VimwikiTokenKind::Tag(_))));
|
||||
}
|
||||
|
||||
// ===== Parser scope rules =====
|
||||
|
||||
#[test]
|
||||
fn scope_is_file_when_tag_is_on_line_0_or_1() {
|
||||
let doc = parse(":todo:other:\n= Heading =\n");
|
||||
let BlockNode::Tag(t) = &doc.children[0] else {
|
||||
panic!(
|
||||
"expected first block to be a Tag, got {:?}",
|
||||
doc.children[0]
|
||||
);
|
||||
};
|
||||
assert_eq!(t.scope, TagScope::File);
|
||||
assert_eq!(t.tags, vec!["todo", "other"]);
|
||||
assert_eq!(doc.metadata.tags, vec!["todo", "other"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scope_is_file_for_tag_on_line_1() {
|
||||
let doc = parse("a paragraph\n:todo:\n");
|
||||
// Tag is on line 1, so still file-level.
|
||||
let tag = doc
|
||||
.children
|
||||
.iter()
|
||||
.find_map(|b| match b {
|
||||
BlockNode::Tag(t) => Some(t),
|
||||
_ => None,
|
||||
})
|
||||
.expect("tag block");
|
||||
assert_eq!(tag.scope, TagScope::File);
|
||||
assert!(doc.metadata.tags.contains(&"todo".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scope_is_heading_when_tag_is_one_or_two_lines_below_heading() {
|
||||
// Heading at line 0; tag at line 1 (one below) → Heading(0).
|
||||
let doc = parse("= H1 =\n:hot:\n");
|
||||
let tag = doc
|
||||
.children
|
||||
.iter()
|
||||
.find_map(|b| match b {
|
||||
BlockNode::Tag(t) => Some(t),
|
||||
_ => None,
|
||||
})
|
||||
.expect("tag block");
|
||||
// Tag at line 1 is *also* line ≤ 1 → file rule wins (matches vimwiki:
|
||||
// file rule applies to lines 0-1 regardless of preceding heading).
|
||||
assert_eq!(tag.scope, TagScope::File);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scope_is_heading_for_tag_two_lines_below_a_deeper_heading() {
|
||||
// Heading at line 3; tag at line 4 (one below).
|
||||
let src = "first\n\n\n= Section =\n:scoped:\n";
|
||||
let doc = parse(src);
|
||||
let tag = doc
|
||||
.children
|
||||
.iter()
|
||||
.find_map(|b| match b {
|
||||
BlockNode::Tag(t) => Some(t),
|
||||
_ => None,
|
||||
})
|
||||
.expect("tag block");
|
||||
// Heading is the only one seen; index 0.
|
||||
assert_eq!(tag.scope, TagScope::Heading(0));
|
||||
// Heading-scope tags do NOT contribute to file-level metadata.
|
||||
assert!(doc.metadata.tags.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scope_is_standalone_for_distant_tag() {
|
||||
// Heading at line 3; tag at line 6 → 3 lines below → out of the
|
||||
// heading window (>2), so standalone.
|
||||
let src = "x\n\n\n= H =\n\n\n:wandering:\n";
|
||||
let doc = parse(src);
|
||||
let tag = doc
|
||||
.children
|
||||
.iter()
|
||||
.find_map(|b| match b {
|
||||
BlockNode::Tag(t) => Some(t),
|
||||
_ => None,
|
||||
})
|
||||
.expect("tag block");
|
||||
assert_eq!(tag.scope, TagScope::Standalone);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scope_is_standalone_when_no_heading_precedes() {
|
||||
// No heading at all and not on lines 0-1.
|
||||
let doc = parse("a\nb\nc\n:lonely:\n");
|
||||
let tag = doc
|
||||
.children
|
||||
.iter()
|
||||
.find_map(|b| match b {
|
||||
BlockNode::Tag(t) => Some(t),
|
||||
_ => None,
|
||||
})
|
||||
.expect("tag block");
|
||||
assert_eq!(tag.scope, TagScope::Standalone);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn file_tags_are_deduped_in_metadata() {
|
||||
let doc = parse(":a:b:\n:b:c:\n");
|
||||
assert_eq!(doc.metadata.tags, vec!["a", "b", "c"]);
|
||||
}
|
||||
|
||||
// ===== Renderer =====
|
||||
|
||||
#[test]
|
||||
fn renderer_emits_div_with_tag_spans() {
|
||||
let html = render(":todo:done:\n");
|
||||
assert!(html.contains("<div class=\"tags\">"));
|
||||
assert!(html.contains("<span class=\"tag\" id=\"tag-todo\">todo</span>"));
|
||||
assert!(html.contains("<span class=\"tag\" id=\"tag-done\">done</span>"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn renderer_escapes_html_inside_tag_names() {
|
||||
// Tags shouldn't contain `<`/`>` per the lexer rule (whitespace
|
||||
// forbidden, but `<` isn't), so this checks the renderer's escaping
|
||||
// fires anyway as defence-in-depth.
|
||||
let html = render(":a<b:\n");
|
||||
assert!(html.contains("a<b"));
|
||||
}
|
||||
@@ -15,7 +15,7 @@ use std::path::{Path, PathBuf};
|
||||
|
||||
use nuwiki_core::ast::{
|
||||
BlockNode, BlockquoteNode, DocumentNode, InlineNode, LinkKind, LinkTarget, ListNode, Span,
|
||||
TableNode,
|
||||
TableNode, TagScope,
|
||||
};
|
||||
use tower_lsp::lsp_types::Url;
|
||||
|
||||
@@ -28,6 +28,19 @@ pub struct IndexedPage {
|
||||
pub title: Option<String>,
|
||||
pub headings: Vec<HeadingInfo>,
|
||||
pub outgoing: Vec<OutgoingLink>,
|
||||
/// Phase 12: every `TagNode` on the page. `tags_by_name` on the
|
||||
/// containing `WorkspaceIndex` is the reverse map.
|
||||
pub tags: Vec<TagInfo>,
|
||||
}
|
||||
|
||||
/// One tag occurrence on a page. `name` is the bare tag string (no
|
||||
/// surrounding colons). Multiple tags on the same source line each get
|
||||
/// their own `TagInfo` so anchor lookup can return a precise location.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct TagInfo {
|
||||
pub name: String,
|
||||
pub scope: TagScope,
|
||||
pub span: Span,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
@@ -56,12 +69,25 @@ pub struct Backlink {
|
||||
pub target_anchor: Option<String>,
|
||||
}
|
||||
|
||||
/// A tag's location, used by `WorkspaceIndex::tags_for` for cross-page
|
||||
/// `[[Page#tag]]` resolution and `nuwiki.tags.search` (Phase 13).
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct TagOccurrence {
|
||||
pub uri: Url,
|
||||
pub span: Span,
|
||||
pub scope: TagScope,
|
||||
}
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
pub struct WorkspaceIndex {
|
||||
pub root: Option<PathBuf>,
|
||||
pub pages_by_uri: HashMap<Url, IndexedPage>,
|
||||
pub pages_by_name: HashMap<String, Url>,
|
||||
pub backlinks: HashMap<String, Vec<Backlink>>,
|
||||
/// Phase 12: tag name → every occurrence across the workspace. Used by
|
||||
/// the v1.1 nav handlers (tag-as-anchor `definition`, `workspace/symbol`
|
||||
/// inclusion) and the Phase 13 `nuwiki.tags.search` command.
|
||||
pub tags_by_name: HashMap<String, Vec<TagOccurrence>>,
|
||||
}
|
||||
|
||||
impl WorkspaceIndex {
|
||||
@@ -84,6 +110,7 @@ impl WorkspaceIndex {
|
||||
title: ast.metadata.title.clone(),
|
||||
headings: Vec::new(),
|
||||
outgoing: Vec::new(),
|
||||
tags: Vec::new(),
|
||||
};
|
||||
index_blocks(&ast.children, &mut page);
|
||||
|
||||
@@ -100,6 +127,18 @@ impl WorkspaceIndex {
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 12: maintain the reverse tag map alongside the per-page list.
|
||||
for tag in &page.tags {
|
||||
self.tags_by_name
|
||||
.entry(tag.name.clone())
|
||||
.or_default()
|
||||
.push(TagOccurrence {
|
||||
uri: uri.clone(),
|
||||
span: tag.span,
|
||||
scope: tag.scope,
|
||||
});
|
||||
}
|
||||
|
||||
self.pages_by_name.insert(name, uri.clone());
|
||||
self.pages_by_uri.insert(uri, page);
|
||||
}
|
||||
@@ -112,6 +151,19 @@ impl WorkspaceIndex {
|
||||
entries.retain(|b| &b.source_uri != uri);
|
||||
}
|
||||
self.backlinks.retain(|_, v| !v.is_empty());
|
||||
for entries in self.tags_by_name.values_mut() {
|
||||
entries.retain(|t| &t.uri != uri);
|
||||
}
|
||||
self.tags_by_name.retain(|_, v| !v.is_empty());
|
||||
}
|
||||
|
||||
/// All occurrences of a tag across the workspace. Empty slice when the
|
||||
/// tag isn't indexed.
|
||||
pub fn tags_for(&self, name: &str) -> &[TagOccurrence] {
|
||||
self.tags_by_name
|
||||
.get(name)
|
||||
.map(Vec::as_slice)
|
||||
.unwrap_or(&[])
|
||||
}
|
||||
|
||||
/// Resolve a `LinkTarget` to a page URI in this workspace.
|
||||
@@ -258,6 +310,15 @@ fn index_block(block: &BlockNode, page: &mut IndexedPage) {
|
||||
}
|
||||
}
|
||||
}
|
||||
BlockNode::Tag(t) => {
|
||||
for name in &t.tags {
|
||||
page.tags.push(TagInfo {
|
||||
name: name.clone(),
|
||||
scope: t.scope,
|
||||
span: t.span,
|
||||
});
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -622,6 +622,24 @@ impl LanguageServer for Backend {
|
||||
});
|
||||
}
|
||||
}
|
||||
// Phase 12: surface tags too, named as `:tag:` so editors
|
||||
// can distinguish them from headings at a glance.
|
||||
for t in &page.tags {
|
||||
if q.is_empty() || t.name.to_lowercase().contains(&q) {
|
||||
#[allow(deprecated)]
|
||||
out.push(LspSymbolInformation {
|
||||
name: format!(":{}:", t.name),
|
||||
kind: SymbolKind::PROPERTY,
|
||||
tags: None,
|
||||
deprecated: None,
|
||||
location: Location {
|
||||
uri: page.uri.clone(),
|
||||
range: span_to_lsp_range_no_text(&t.span),
|
||||
},
|
||||
container_name: Some(page.name.clone()),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(Some(out))
|
||||
@@ -964,8 +982,15 @@ fn zero_range() -> Range {
|
||||
|
||||
fn heading_range_in(idx: &WorkspaceIndex, uri: &Url, anchor: &str, _utf8: bool) -> Option<Range> {
|
||||
let page = idx.page(uri)?;
|
||||
let h = page.headings.iter().find(|h| h.anchor == anchor)?;
|
||||
Some(span_to_lsp_range_no_text(&h.span))
|
||||
// Headings win over tags when both match — same precedence as
|
||||
// vimwiki: heading anchors are the "canonical" target shape.
|
||||
if let Some(h) = page.headings.iter().find(|h| h.anchor == anchor) {
|
||||
return Some(span_to_lsp_range_no_text(&h.span));
|
||||
}
|
||||
// Phase 12: tags-as-anchors. `[[Page#tag-name]]` lands on the
|
||||
// matching `TagNode` on the target page.
|
||||
let t = page.tags.iter().find(|t| t.name == anchor)?;
|
||||
Some(span_to_lsp_range_no_text(&t.span))
|
||||
}
|
||||
|
||||
fn is_inside_wikilink(text: &str, line: u32, byte_col: u32) -> bool {
|
||||
|
||||
@@ -46,6 +46,7 @@ pub const TOKEN_TRANSCLUSION: u32 = 15;
|
||||
pub const TOKEN_URL: u32 = 16;
|
||||
pub const TOKEN_COMMENT: u32 = 17;
|
||||
pub const TOKEN_DEFINITION_TERM: u32 = 18;
|
||||
pub const TOKEN_TAG: u32 = 19;
|
||||
|
||||
pub const TOKEN_TYPE_NAMES: &[&str] = &[
|
||||
"vimwikiHeading",
|
||||
@@ -67,6 +68,7 @@ pub const TOKEN_TYPE_NAMES: &[&str] = &[
|
||||
"vimwikiUrl",
|
||||
"vimwikiComment",
|
||||
"vimwikiDefinitionTerm",
|
||||
"vimwikiTag",
|
||||
];
|
||||
|
||||
pub const MOD_LEVEL1: u32 = 1 << 0;
|
||||
@@ -197,6 +199,7 @@ fn emit_block(block: &BlockNode, text: &str, idx: &LineIndex, out: &mut Vec<RawT
|
||||
}
|
||||
}
|
||||
BlockNode::Comment(c) => split_span(c.span, text, idx, TOKEN_COMMENT, 0, out),
|
||||
BlockNode::Tag(t) => split_span(t.span, text, idx, TOKEN_TAG, 0, out),
|
||||
BlockNode::Error(_) => {}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
//! Phase 12 LSP-side tag tests:
|
||||
//! - WorkspaceIndex maintains per-page tag lists and a reverse map.
|
||||
//! - `tags_for` returns occurrences across pages.
|
||||
//! - `upsert` replaces stale tags on re-parse.
|
||||
//! - `remove` drops the page's tags + cleans `tags_by_name`.
|
||||
|
||||
use std::path::PathBuf;
|
||||
|
||||
use nuwiki_core::syntax::vimwiki::VimwikiSyntax;
|
||||
use nuwiki_core::syntax::SyntaxPlugin;
|
||||
use nuwiki_lsp::index::WorkspaceIndex;
|
||||
use tower_lsp::lsp_types::Url;
|
||||
|
||||
fn parse(src: &str) -> nuwiki_core::ast::DocumentNode {
|
||||
VimwikiSyntax::new().parse(src)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn upsert_populates_per_page_tags() {
|
||||
let mut idx = WorkspaceIndex::new(Some(PathBuf::from("/wiki")));
|
||||
let uri = Url::from_file_path("/wiki/Home.wiki").unwrap();
|
||||
idx.upsert(uri.clone(), &parse(":alpha:beta:\n= Home =\n"));
|
||||
|
||||
let page = idx.page(&uri).expect("indexed");
|
||||
assert_eq!(page.tags.len(), 2);
|
||||
let names: Vec<_> = page.tags.iter().map(|t| t.name.as_str()).collect();
|
||||
assert!(names.contains(&"alpha"));
|
||||
assert!(names.contains(&"beta"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tags_by_name_aggregates_across_pages() {
|
||||
let mut idx = WorkspaceIndex::new(Some(PathBuf::from("/wiki")));
|
||||
let home = Url::from_file_path("/wiki/Home.wiki").unwrap();
|
||||
let about = Url::from_file_path("/wiki/About.wiki").unwrap();
|
||||
idx.upsert(home.clone(), &parse(":shared:\n"));
|
||||
idx.upsert(about.clone(), &parse(":shared:other:\n"));
|
||||
|
||||
let shared = idx.tags_for("shared");
|
||||
assert_eq!(shared.len(), 2);
|
||||
let uris: Vec<&Url> = shared.iter().map(|o| &o.uri).collect();
|
||||
assert!(uris.contains(&&home));
|
||||
assert!(uris.contains(&&about));
|
||||
|
||||
assert_eq!(idx.tags_for("other").len(), 1);
|
||||
assert!(idx.tags_for("nonexistent").is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn re_upsert_replaces_stale_tags() {
|
||||
let mut idx = WorkspaceIndex::new(Some(PathBuf::from("/wiki")));
|
||||
let uri = Url::from_file_path("/wiki/Home.wiki").unwrap();
|
||||
idx.upsert(uri.clone(), &parse(":old:\n"));
|
||||
assert_eq!(idx.tags_for("old").len(), 1);
|
||||
|
||||
idx.upsert(uri.clone(), &parse(":new:\n"));
|
||||
assert!(idx.tags_for("old").is_empty());
|
||||
assert_eq!(idx.tags_for("new").len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn remove_drops_tags_and_reverse_entries() {
|
||||
let mut idx = WorkspaceIndex::new(Some(PathBuf::from("/wiki")));
|
||||
let uri = Url::from_file_path("/wiki/Home.wiki").unwrap();
|
||||
idx.upsert(uri.clone(), &parse(":a:b:\n"));
|
||||
idx.remove(&uri);
|
||||
|
||||
assert!(idx.page(&uri).is_none());
|
||||
assert!(idx.tags_for("a").is_empty());
|
||||
assert!(idx.tags_for("b").is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn heading_scope_tag_indexed_but_not_in_metadata() {
|
||||
// Tag at the third line below a heading is `Heading`-scoped; the
|
||||
// metadata.tags accumulator only collects file-scope tags.
|
||||
let mut idx = WorkspaceIndex::new(Some(PathBuf::from("/wiki")));
|
||||
let uri = Url::from_file_path("/wiki/Home.wiki").unwrap();
|
||||
let doc = parse("para\n\n\n= H =\n:scoped:\n");
|
||||
idx.upsert(uri.clone(), &doc);
|
||||
|
||||
assert_eq!(idx.tags_for("scoped").len(), 1);
|
||||
assert!(doc.metadata.tags.is_empty());
|
||||
}
|
||||
Reference in New Issue
Block a user