phase 12: tags — lexer, AST, index, anchor resolution, ws/symbol
Vimwiki `:tag1:tag2:` syntax across the full stack. P11 resolved as
strict-vimwiki — no markdown `#tag` alias.
AST (nuwiki-core):
- `TagScope { File, Heading(usize), Standalone }` mirrors vimwiki's
three placement-determined meanings: lines 0-1 → File; one or two
lines below the most recent heading → Heading(idx); otherwise
Standalone.
- `TagNode { span, tags: Vec<String>, scope }` + `BlockNode::Tag`.
- `PageMetadata.tags: Vec<String>` accumulates the file-scope tag
names — convenience projection of the Tag blocks whose scope is
File. New field is additive (Default derive) so existing consumers
using `..Default::default()` keep working.
- `Visitor::visit_tag` lands with a default body and `walk_block`
dispatches the new arm.
Lexer (`syntax/vimwiki/lexer.rs`):
- `VimwikiTokenKind::Tag(Vec<String>)` block-level token.
- `try_lex_tag` runs in the normal-line dispatcher right after heading
recognition; demands the *whole trimmed line* match `:name:name…:`,
with non-empty whitespace-free names between adjacent colons. Empty
segments (`::tag::`), whitespace inside a name, or single-colon
lines are all rejected. Doesn't fight `Term::` (definition-list
marker) because that pattern requires text before the `::`.
- Indentation is tolerated; the span covers the trimmed run.
Parser (`syntax/vimwiki/parser.rs`):
- `ParseState` gains `headings_seen` + `last_heading_line`.
- `parse_heading` bumps both on every closed heading so the tag
parser can detect the heading window.
- `parse_tag` reads the Tag token, computes the scope via
`scope_for_tag(tag_line)`, eats the trailing newline, and emits the
TagNode.
- `parse_document` post-processes each block: when a Tag with
`scope == File` is emitted, its names are pushed (de-duped) into
`metadata.tags`.
Renderer (`render/html.rs`):
- `render_tag` emits `<div class="tags"><span class="tag"
id="tag-name">name</span>...</div>`. The `id` is what makes
`[[Page#tag-name]]` jump to the rendered tag.
Semantic tokens (`semantic_tokens.rs`):
- New `TOKEN_TAG = 19` / `vimwikiTag` so editors highlight tag lines
distinctly. `emit_block` adds the Tag arm.
Workspace index (`index.rs`):
- `IndexedPage.tags: Vec<TagInfo>` records each tag occurrence on a
page with its span + scope.
- `WorkspaceIndex.tags_by_name: HashMap<String, Vec<TagOccurrence>>`
is the reverse map for cross-page anchor resolution + the Phase 13
`nuwiki.tags.search` command.
- `upsert` populates both; `remove` scrubs the reverse map and drops
empty entries; `tags_for(name)` is the public lookup.
LSP handlers (`lib.rs`):
- `heading_range_in` falls back to tag matching when the anchor
isn't a heading slug — so `[[Page#some-tag]]` lands on the matching
TagNode. Heading anchors win over tag anchors when both match (the
vimwiki precedence).
- `workspace/symbol` includes tags alongside headings, named `:tag:`
and kinded `PROPERTY` so editor UIs can distinguish them.
Tests (20 new):
- Syntax (15 in `vimwiki_tags.rs`):
lexer recognition + rejection of `::tag::`, whitespace-in-name,
Term::-conflict; parser scope paths (file, heading-1-below,
heading-2-below, standalone, no-heading, line-1 priority);
metadata dedup; renderer HTML + escaping.
- LSP index (5 in `tags_index_and_lsp.rs`):
per-page tag population, cross-page `tags_by_name` aggregation,
re-upsert replaces stale tags, `remove` cleans the reverse map,
heading-scope tag indexed but not in file metadata.
All 191 v1.0 + Phase 11 tests still pass — backward compatible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -39,7 +39,7 @@ use crate::ast::{
|
||||
InlineNode, ItalicNode, KeywordNode, LinkKind, LinkTarget, ListItemNode, ListNode, ListSymbol,
|
||||
MathBlockNode, MathInlineNode, PageMetadata, ParagraphNode, PreformattedNode, RawUrlNode, Span,
|
||||
StrikethroughNode, SubscriptNode, SuperscriptNode, TableCellNode, TableNode, TableRowNode,
|
||||
TextNode, TransclusionNode, WikiLinkNode,
|
||||
TagNode, TagScope, TextNode, TransclusionNode, WikiLinkNode,
|
||||
};
|
||||
use crate::syntax::{Parser, TokenStream};
|
||||
|
||||
@@ -69,11 +69,22 @@ impl Parser for VimwikiParser {
|
||||
struct ParseState<'a> {
|
||||
toks: &'a [VimwikiToken],
|
||||
pos: usize,
|
||||
/// Phase 12: count of `HeadingNode`s already pushed to `children`.
|
||||
/// `headings_seen - 1` is the most recent heading's index.
|
||||
headings_seen: usize,
|
||||
/// Phase 12: source line of the most recently-emitted heading.
|
||||
/// `tag_line - last_heading_line ∈ {1, 2}` is a header-scope tag.
|
||||
last_heading_line: Option<u32>,
|
||||
}
|
||||
|
||||
impl<'a> ParseState<'a> {
|
||||
fn new(toks: &'a [VimwikiToken]) -> Self {
|
||||
Self { toks, pos: 0 }
|
||||
Self {
|
||||
toks,
|
||||
pos: 0,
|
||||
headings_seen: 0,
|
||||
last_heading_line: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn at_eof(&self) -> bool {
|
||||
@@ -129,7 +140,21 @@ impl<'a> ParseState<'a> {
|
||||
continue;
|
||||
}
|
||||
let before = self.pos;
|
||||
children.push(self.parse_block());
|
||||
let block = self.parse_block();
|
||||
// Phase 12: file-scope tags get accumulated into metadata so
|
||||
// callers can read page-level tag membership without walking
|
||||
// the AST. The TagNode itself stays in `children` so renderers
|
||||
// see it too.
|
||||
if let BlockNode::Tag(t) = &block {
|
||||
if t.scope == TagScope::File {
|
||||
for name in &t.tags {
|
||||
if !metadata.tags.iter().any(|t| t == name) {
|
||||
metadata.tags.push(name.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
children.push(block);
|
||||
// Resilience: guarantee forward progress.
|
||||
if self.pos == before {
|
||||
let span = self.toks.get(self.pos).map(|t| t.span).unwrap_or_default();
|
||||
@@ -190,6 +215,7 @@ impl<'a> ParseState<'a> {
|
||||
K::MathBlockOpen { .. } => self.parse_math_block(),
|
||||
K::CommentLine(_) => self.parse_single_comment(),
|
||||
K::CommentMultiOpen => self.parse_multi_comment(),
|
||||
K::Tag(_) => self.parse_tag(),
|
||||
_ => {
|
||||
if self.is_definition_start() {
|
||||
self.parse_definition_list()
|
||||
@@ -200,6 +226,42 @@ impl<'a> ParseState<'a> {
|
||||
}
|
||||
}
|
||||
|
||||
// ===== Tag =====
|
||||
|
||||
fn parse_tag(&mut self) -> BlockNode {
|
||||
let t = self.advance().unwrap();
|
||||
let names = match &t.kind {
|
||||
K::Tag(v) => v.clone(),
|
||||
_ => unreachable!(),
|
||||
};
|
||||
let span = t.span;
|
||||
let scope = self.scope_for_tag(span.start.line);
|
||||
self.eat_newline();
|
||||
BlockNode::Tag(TagNode {
|
||||
span,
|
||||
tags: names,
|
||||
scope,
|
||||
})
|
||||
}
|
||||
|
||||
/// Decide the placement-determined scope of a tag at the given source
|
||||
/// line, matching vimwiki's rules:
|
||||
/// - lines 0 or 1 → `File`
|
||||
/// - within two lines after the most recent heading → `Heading(idx)`
|
||||
/// - otherwise → `Standalone`
|
||||
fn scope_for_tag(&self, tag_line: u32) -> TagScope {
|
||||
if tag_line <= 1 {
|
||||
return TagScope::File;
|
||||
}
|
||||
if let Some(h_line) = self.last_heading_line {
|
||||
let delta = tag_line.saturating_sub(h_line);
|
||||
if (1..=2).contains(&delta) && self.headings_seen > 0 {
|
||||
return TagScope::Heading(self.headings_seen - 1);
|
||||
}
|
||||
}
|
||||
TagScope::Standalone
|
||||
}
|
||||
|
||||
// ===== Heading =====
|
||||
|
||||
fn parse_heading(&mut self) -> BlockNode {
|
||||
@@ -218,6 +280,9 @@ impl<'a> ParseState<'a> {
|
||||
let inline = parse_inline_seq(&self.toks[inline_start..self.pos]);
|
||||
self.advance();
|
||||
self.eat_newline();
|
||||
// Record for tag-scope resolution (Phase 12).
|
||||
self.headings_seen += 1;
|
||||
self.last_heading_line = Some(span_end.line);
|
||||
return BlockNode::Heading(HeadingNode {
|
||||
span: Span::new(span_start, span_end),
|
||||
level,
|
||||
@@ -814,6 +879,7 @@ fn starts_new_block(kind: &K) -> bool {
|
||||
| K::PlaceholderNohtml
|
||||
| K::PlaceholderTemplate(_)
|
||||
| K::PlaceholderDate(_)
|
||||
| K::Tag(_)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user