9d34c8baa25794373e11e79824d47d6c54ddfb45
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 gffranco
|
8b29d4e9b9 |
phase 12: tags — lexer, AST, index, anchor resolution, ws/symbol
Vimwiki `:tag1:tag2:` syntax across the full stack. P11 resolved as
strict-vimwiki — no markdown `#tag` alias.
AST (nuwiki-core):
- `TagScope { File, Heading(usize), Standalone }` mirrors vimwiki's
three placement-determined meanings: lines 0-1 → File; one or two
lines below the most recent heading → Heading(idx); otherwise
Standalone.
- `TagNode { span, tags: Vec<String>, scope }` + `BlockNode::Tag`.
- `PageMetadata.tags: Vec<String>` accumulates the file-scope tag
names — convenience projection of the Tag blocks whose scope is
File. New field is additive (Default derive) so existing consumers
using `..Default::default()` keep working.
- `Visitor::visit_tag` lands with a default body and `walk_block`
dispatches the new arm.
Lexer (`syntax/vimwiki/lexer.rs`):
- `VimwikiTokenKind::Tag(Vec<String>)` block-level token.
- `try_lex_tag` runs in the normal-line dispatcher right after heading
recognition; demands the *whole trimmed line* match `:name:name…:`,
with non-empty whitespace-free names between adjacent colons. Empty
segments (`::tag::`), whitespace inside a name, or single-colon
lines are all rejected. Doesn't fight `Term::` (definition-list
marker) because that pattern requires text before the `::`.
- Indentation is tolerated; the span covers the trimmed run.
Parser (`syntax/vimwiki/parser.rs`):
- `ParseState` gains `headings_seen` + `last_heading_line`.
- `parse_heading` bumps both on every closed heading so the tag
parser can detect the heading window.
- `parse_tag` reads the Tag token, computes the scope via
`scope_for_tag(tag_line)`, eats the trailing newline, and emits the
TagNode.
- `parse_document` post-processes each block: when a Tag with
`scope == File` is emitted, its names are pushed (de-duped) into
`metadata.tags`.
Renderer (`render/html.rs`):
- `render_tag` emits `<div class="tags"><span class="tag"
id="tag-name">name</span>...</div>`. The `id` is what makes
`[[Page#tag-name]]` jump to the rendered tag.
Semantic tokens (`semantic_tokens.rs`):
- New `TOKEN_TAG = 19` / `vimwikiTag` so editors highlight tag lines
distinctly. `emit_block` adds the Tag arm.
Workspace index (`index.rs`):
- `IndexedPage.tags: Vec<TagInfo>` records each tag occurrence on a
page with its span + scope.
- `WorkspaceIndex.tags_by_name: HashMap<String, Vec<TagOccurrence>>`
is the reverse map for cross-page anchor resolution + the Phase 13
`nuwiki.tags.search` command.
- `upsert` populates both; `remove` scrubs the reverse map and drops
empty entries; `tags_for(name)` is the public lookup.
LSP handlers (`lib.rs`):
- `heading_range_in` falls back to tag matching when the anchor
isn't a heading slug — so `[[Page#some-tag]]` lands on the matching
TagNode. Heading anchors win over tag anchors when both match (the
vimwiki precedence).
- `workspace/symbol` includes tags alongside headings, named `:tag:`
and kinded `PROPERTY` so editor UIs can distinguish them.
Tests (20 new):
- Syntax (15 in `vimwiki_tags.rs`):
lexer recognition + rejection of `::tag::`, whitespace-in-name,
Term::-conflict; parser scope paths (file, heading-1-below,
heading-2-below, standalone, no-heading, line-1 priority);
metadata dedup; renderer HTML + escaping.
- LSP index (5 in `tags_index_and_lsp.rs`):
per-page tag population, cross-page `tags_by_name` aggregation,
re-upsert replaces stale tags, `remove` cleans the reverse map,
heading-scope tag indexed but not in file metadata.
All 191 v1.0 + Phase 11 tests still pass — backward compatible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
gffranco
|
f00b0780b8 |
phase 5: renderer trait + HtmlRenderer
Writer-based `Renderer` trait per SPEC §6.8: object-safe
`fn render(&self, doc, w: &mut dyn Write) -> io::Result<()>` plus
a `render_to_string` convenience. Object safety preserves the option
of `Box<dyn Renderer>` in the LSP layer.
`HtmlRenderer` builder accepts a link-resolution callback (via
`with_link_resolver`) and an optional enclosing template (via
`with_template`). The default resolver mirrors §9: Wiki paths become
`path.html`, interwiki links land in sibling directories
(`../wiki<N>/` / `../wn-<Name>/`), diary entries under `diary/`,
file/local schemes use literal paths, anchor-only collapses to `#`.
Renderer covers every AST node:
- block: heading (centered class), paragraph, hr, blockquote,
preformatted (`language-<lang>` class), math block (with
`\begin{env}` when an environment is set), lists (ordered/unordered
+ every checkbox state, plus recursive sublists), definition list
(dt/dd), table (thead/tbody driven by `is_header`, col/row span as
CSS class hooks), comment (as HTML comment with `--` neutralised),
error node (as `.parse-error` span)
- inline: strong/em + bold-italic, `<del>`, code (escaped),
`<sup>`/`<sub>`, math-inline wrapped in `\(...\)`, keyword spans
with per-keyword class, color spans, wikilink/external/raw URL
anchors, transclusion as `<img>` with sorted attrs
HTML escaping is byte-level (`<>&"'`). Template substitution replaces
`{{content}}` first so a body containing the literal `{{title}}`
isn't double-substituted.
Tests (31 new) cover every block + inline construct, every LinkKind
through the default resolver, HTML escaping, custom resolver
override, template with/without title metadata, and an end-to-end
smoke test on a multi-construct document.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
gffranco
|
294963b517 |
phase 4: vimwiki parser + SyntaxPlugin glue
Hand-rolled recursive-descent parser over the token stream. Resilient per SPEC §6.7: never aborts the document — anything the dispatcher can't claim becomes BlockNode::Error and the cursor advances, and unterminated wikilinks / transclusions / delimiters fall back to literal text. Block coverage: every variant from §6.4 — Heading (with inline content between markers), Paragraph (with soft-break across single newlines), HorizontalRule, Blockquote (both `>` and 4-space styles), Preformatted, MathBlock, List (with checkbox + indent-driven sublists), DefinitionList, Table (header separator promotes the preceding row, plus colspan / rowspan cells), Comment (single + multiline), and page-level placeholders threaded into PageMetadata. Inline pairing: left-to-right scan with "find next matching delim", recursing on the slice between. `_*x*_` becomes Italic(Bold(...)), mirroring SPEC §6.7 precedence. URLs inside `[[ ]]` route to ExternalLinkNode; everything else to WikiLinkNode with full LinkTarget classification (Wiki / Interwiki numbered + named / Diary / File / Local / AnchorOnly / directory / root-relative / filesystem-absolute). VimwikiSyntax registers as id="vimwiki", extensions=[".wiki"] and chains lexer + parser inside SyntaxPlugin::parse. SPEC §4 updated to record the hand-rolled choice with rationale — span-bearing tokens fit awkwardly into winnow/nom combinator style; resilience and recovery are explicit in code instead. Tests (41 new parser cases) cover every AST construct, inline precedence including nested bold/italic, every LinkKind, transclusion alt + attrs, resilience on unterminated links, and end-to-end SyntaxRegistry dispatch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
gffranco
|
942dbe2aa8 |
phase 3: vimwiki lexer
Two-pass, hand-rolled lexer per SPEC.md §6.6. Block pass walks the
source line by line; recognised constructs emit structural tokens, and
text-bearing line content is fed to the inline pass. Multi-line fences
(`{{{ }}}`, `{{$ }}$`, `%%+ +%%`) flip a `BlockMode` so subsequent
lines are captured raw until the matching closer.
VimwikiToken covers the full §9 feature checklist:
- headings (1–6, centered), horizontal rule, blockquotes (`>` and
4-space), all list-marker variants, checkboxes, definition `::`,
tables (separators, header row, col/row span)
- preformatted fences (with language + key=val attrs), math blocks
(with `%env%`), single- and multi-line comments, all four
page placeholders (%title / %nohtml / %template / %date)
- inline: bold/italic/strike/super/sub delimiters, inline code, inline
math, all six keywords (word-bounded), wikilinks (with description
separator), transclusions (with attr separator), raw URLs across
http(s)/ftp/mailto/file schemes — trailing sentence punctuation
stripped from URL spans
Spans are byte-accurate per SPEC §6.5 (0-indexed line + byte column +
absolute byte offset). The lexer is permissive: every delimiter is
emitted, the parser will pair them and fall back to literal text on
mismatches.
Tests (41 cases) cover one example per construct in §9, plus
word-boundary keyword detection, list-vs-bold disambiguation,
indent-vs-list disambiguation, and a span-correctness check.
README now carries a per-phase status table; SPEC top-line status
advanced to Phase 3 complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
gffranco
|
8f35c0a80c |
phase 2: syntax plugin interface and registry
Add the Lexer/Parser/SyntaxPlugin traits and the SyntaxRegistry per SPEC.md §6.3. - `Lexer<Token>` and `Parser<Token>` are typed building blocks: each syntax owns its own token alphabet (vimwiki today, markdown later). - `TokenStream<T>` is a `Vec<T>` newtype so the eager-vs-lazy choice (§6.6) can flip later without breaking callers. - `SyntaxPlugin` is type-erased: it exposes id / display_name / file_extensions / parse(text) so the registry can hold heterogeneous plugins behind a single trait object. Implementations chain their Lexer + Parser inside parse(). - `SyntaxPlugin: Send + Sync` so the LSP server can stash plugins in an Arc and share them across handler tasks. - `SyntaxRegistry` stores `Arc<dyn SyntaxPlugin>`, supports lookup by id and by file extension, iteration in registration order. Tests cover TokenStream round-trip, full lex→parse chain through a mock plugin, registry lookup hits and misses, and a compile-time assert_send_sync::<SyntaxRegistry>(). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
gffranco
|
dc2a952e67 |
phase 1: core AST, spans, and Visitor
Define every node type in SPEC.md §6.4 — Document, 11 block variants, 15 inline variants (including the 4 link-related ones), plus ListItem / DefinitionItem / TableRow / TableCell, ListSymbol, CheckboxState, Keyword, LinkTarget, LinkKind. Every node carries a `Span` (line + column + byte offset, 0-indexed) per §6.5. Visitor: open-recursion trait with default `visit_*` bodies that delegate to `walk_*` free functions. Override at any granularity; call `walk_*` from your override to keep descending. P4 resolved by defining the trait now so Phase 5 (renderer) and Phase 7 (semantic tokens) can plug in without refactoring traversal. P3 resolved: AST string fields are owned `String`. Cow / Arc<str> can be revisited if the parser grows a zero-copy mode. Tests: visitor smoke test builds a heading + paragraph + list + table by hand and asserts exact visit counts (4 blocks, 12 inlines, 9 text leaves, 1 bold, 1 external link, 1 transclusion). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |