Two-pass, hand-rolled lexer per SPEC.md §6.6. Block pass walks the
source line by line; recognised constructs emit structural tokens, and
text-bearing line content is fed to the inline pass. Multi-line fences
(`{{{ }}}`, `{{$ }}$`, `%%+ +%%`) flip a `BlockMode` so subsequent
lines are captured raw until the matching closer.
VimwikiToken covers the full §9 feature checklist:
- headings (1–6, centered), horizontal rule, blockquotes (`>` and
4-space), all list-marker variants, checkboxes, definition `::`,
tables (separators, header row, col/row span)
- preformatted fences (with language + key=val attrs), math blocks
(with `%env%`), single- and multi-line comments, all four
page placeholders (%title / %nohtml / %template / %date)
- inline: bold/italic/strike/super/sub delimiters, inline code, inline
math, all six keywords (word-bounded), wikilinks (with description
separator), transclusions (with attr separator), raw URLs across
http(s)/ftp/mailto/file schemes — trailing sentence punctuation
stripped from URL spans
Spans are byte-accurate per SPEC §6.5 (0-indexed line + byte column +
absolute byte offset). The lexer is permissive: every delimiter is
emitted, the parser will pair them and fall back to literal text on
mismatches.
Tests (41 cases) cover one example per construct in §9, plus
word-boundary keyword detection, list-vs-bold disambiguation,
indent-vs-list disambiguation, and a span-correctness check.
README now carries a per-phase status table; SPEC top-line status
advanced to Phase 3 complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add the Lexer/Parser/SyntaxPlugin traits and the SyntaxRegistry per
SPEC.md §6.3.
- `Lexer<Token>` and `Parser<Token>` are typed building blocks: each
syntax owns its own token alphabet (vimwiki today, markdown later).
- `TokenStream<T>` is a `Vec<T>` newtype so the eager-vs-lazy choice
(§6.6) can flip later without breaking callers.
- `SyntaxPlugin` is type-erased: it exposes id / display_name /
file_extensions / parse(text) so the registry can hold heterogeneous
plugins behind a single trait object. Implementations chain their
Lexer + Parser inside parse().
- `SyntaxPlugin: Send + Sync` so the LSP server can stash plugins in
an Arc and share them across handler tasks.
- `SyntaxRegistry` stores `Arc<dyn SyntaxPlugin>`, supports lookup by
id and by file extension, iteration in registration order.
Tests cover TokenStream round-trip, full lex→parse chain through a
mock plugin, registry lookup hits and misses, and a compile-time
assert_send_sync::<SyntaxRegistry>().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gffranco