Fix navigation: level-based chunking, nested toc.yml, discrete sections, navigation_title - #3833
Open
Mpdreamz wants to merge 31 commits into
Open
Fix navigation: level-based chunking, nested toc.yml, discrete sections, navigation_title#3833Mpdreamz wants to merge 31 commits into
Mpdreamz wants to merge 31 commits into
Conversation
Implements a native C# converter to migrate elastic.co/guide content (AsciiDoc served by the legacy elastic/docs system) into Markdown docsets that docs-builder can render. New project: Elastic.LegacyDocs.Migration with: - conf.yaml parser for the elastic/docs book definitions - Source repo manager for resolving branches and paths - AsciiDoc lexer, parser, and AST (sections, code blocks, tables, admonitions, lists, includes, conditionals, inline formatting) - Markdown emitter targeting MyST syntax - Page chunker to split documents by section level - Archive and Latest docset generators producing docset.yml/toc.yml - CLI command: `docs-builder guide migrate` Made-with: Cursor
Fixes found by running the converter end-to-end and building with docs-builder: - Fix section level off-by-one in lexer (= is level 0, == is level 1) - Fix Repos model to handle nested YAML objects (Dict<string, LegacyRepo>) - Fix basePath in generators to use index file directory for includes - Fix chunker to walk into level-0 sections and promote chunked sections to H1 for standalone pages - Use docs-builder anchor syntax (# Title [anchor]) instead of MyST (anchor)= targets - Map unsupported directives: caution→warning, sidebar→admonition - Add cross-page anchor-to-slug map so refs emit file links - Add bare URL[text] pattern to inline parser - Prevent double image path prefix Made-with: Cursor
- Add DSV/CSV/TSV table format support (384 occurrences in ES|QL docs) - Add pass:[] and +inline+ passthrough inline patterns - Add unconstrained bold (**text**) support - Add [role]#text# inline macro support - Extract migration command into standalone docs-migrate CLI tool (removable by deleting src/tooling/docs-migrate/ and src/authoring/Elastic.LegacyDocs.Migration/) - Remove migration coupling from docs-builder Made-with: Cursor
…ktree strategy - Split monolithic MigrateCommand into init/list/clone/convert/serve subcommands - SourceRepoManager now uses bare clones + git worktrees + sparse checkout (cone mode) to avoid redundant full clones per branch and only fetch needed directory trees - Add Proc package for process execution in place of raw System.Diagnostics.Process - Add guide-nav feature flag to configuration layer (FeatureFlags, DocumentationSetFile, ConfigurationFile) so converted docsets can signal the guide-archive nav mode Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The conf.yaml structure has up to 3 levels of nesting: a category's sections can contain sub-groups (with base_dir + sections) rather than books directly. LegacyBook had no Sections property so IgnoreUnmatchedProperties silently dropped all nested books, causing entire families to disappear (Elasticsearch clients, all APM agents, ECS logging, etc.). Add BaseDir and Sections to LegacyBook, then recursively flatten after deserialization, accumulating the base_dir prefix so leaf book prefixes are fully qualified (e.g. en/elasticsearch/client/net-api). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Use LocalApplicationData/elastic/docs-migrate as the default work directory, matching the pattern from Paths.ApplicationData in the main codebase. Includes the same Docker/CI fallback to GetTempPath() when LocalApplicationData returns an empty string. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Output from convert and serve now lands in .artifacts/migrated relative to the current working directory, separate from the AppData work dir that holds conf.yaml and repo clones. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t versioned branches - Add --no-hud to docs-builder serve: disables diagnostics HUD and skips background in-memory validation builds - docs-migrate serve passes --no-hud automatically - Fix SourceRepoManager branch resolution: check local branches before fetching, use --detach for worktrees so repos like elastic/docs (only has master) can serve multiple version worktrees - Use Proc.Start for branch probing to suppress expected git errors Made-with: Cursor
- Rewrite https://www.elastic.co/guide/... URLs to absolute internal markdown links (/prefix/version/page.md) so docs-builder validates them - Emit {{name}} (double braces) for unresolved AsciiDoc attribute references so docs-builder recognizes them as substitution variables - Add SharedAttributes with ~130 well-known product-name attributes from shared/attributes.asciidoc (es, kib, agent, fleet, etc.) - Write subs: section in generated docset.yml with all product names Made-with: Cursor
AsciiDoc [[id]] anchors on non-section blocks (paragraphs, tables, description lists, etc.) were silently discarded. Now they are wrapped in an AnchoredBlock node and emitted as $$$id$$$ inline anchors that docs-builder renders as <a id="..."></a> link targets. Made-with: Cursor
…rce attributes - Fix lexer: conditionals inside verbatim/table blocks were being parsed as conditional tokens instead of raw text; now only recognized outside those contexts - Fix table delimiter regex to accept |=== with 3+ = signs - Fix parser: attribute substitution in section titles and image alt/title; preserve pending id/title/blockAttr across list-continuation blocks; track pendingStart to reset pos on section-level break - Fix page chunker: include AnchoredBlock IDs in anchor-to-slug map - Fix ConvertCommand: pass source_branch and per-repo root attributes to parser; guard against missing Current version when building overview Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
docs-migrate was the last CLI still on ConsoleAppFramework, which had no PackageVersion entry and so failed restore under Central Package Management. Align it with docs-builder and essc: Host.CreateApplicationBuilder + AddArgh, app.Map<T>() registration, and CancellationToken in place of Cancel. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes eight root-cause bugs that produced garbled output across 21,962+
files in the converted corpus. All changes are covered by a new
tests/Elastic.LegacyDocs.Migration.Tests/ project (19 tests).
Step 0: Add test project, diagnostics channel (OnDiagnostic on
AsciidocParserOptions), and AGENTS.md test-project table row.
Step 1: Fix `--` open blocks — lexer dispatched on delimChar before
checking length so `--` fell into the verbatim path. Now length < 4
routes to BlockDelimiter; IsMatchingDelimiter/IsMatchingClose require
exact length so `--------` can no longer close a `----` block.
PageChunker now recurses into OpenBlockNode for anchor collection and
page extraction.
Step 2: Add IncludeDirective case to ParseBlock's switch — includes
were silently skipped (SkipToken) instead of being processed.
Step 3: Attribute resolution — eager expansion via SetAttribute helper,
SetBasePath keeps docdir in sync, Path.GetFullPath normalization in
ProcessInclude fixes 863 `../../shared/attributes.asciidoc` paths,
docs-root/asciidoc-dir seeded in ConvertCommand, docdir saved and
restored across include file boundaries.
Step 4: Emitter substitution — undefined attributes emit `{name}` not
`{{name}}`; ProductNames keys pass through as `{{name}}` for
docs-builder subs. SetAttribute skips ProductNames so shared/
attributes.asciidoc cannot shadow them.
Step 5: Callout annotations — populate CodeBlockNode.Callouts from
trailing `<n> text` lines after a code block closing delimiter,
making the emitter's ordered-list output live.
Step 6: include-tagged:: — new ResolveVerbatimIncludes handles
`include-tagged::` directives inside listing blocks; tag/end regexes
widened to allow hyphens and drop the //prefix requirement.
Step 7: Xref regex uses non-greedy `.+?` to allow `>` in link text;
multi-line admonition paragraphs collect continuation Text tokens
instead of truncating at the first line.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Headings: apply attribute substitution to document/section titles so
{es} → {{es}} rather than leaking as raw text
- Same-page xrefs: compare resolved anchor slug against current page
slug; emit #anchor for same-page and slug.md#anchor for cross-page
- AnchoredBlock: emit a blank line after $$$anchor$$$ so docs-builder
treats it as a standalone inline anchor block
- titleabbrev: PassthroughNode wrapping <titleabbrev>…</titleabbrev>
was missed by the ParagraphNode filter; add the check to the
PassthroughNode emit branch
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add --minors N to clone and convert commands to cap minor versions per major; stored in FilterOptions and persisted to .clone-options.json - Change docs-migrate serve default port from 3000 to 3001 - Fix /8.19 URL routing: Path.GetExtension treats version segments like 8.19 as having extension .19; restrict hasKnownExtension check to actual document file types so version-number slugs resolve correctly Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Build AnchorToTitleMap alongside AnchorToSlugMap in PageChunker;
when a section has <titleabbrev>, use that as the display title
(matches the abbreviated nav title on elastic.co, e.g. "Indices and
documents" vs full "Indices, documents, and fields")
- EmitCrossRef: resolve <<anchor>> with no text to the section title
from AnchorToTitleMap, not the raw anchor ID
- EmitLink: apply SubstituteTitleAttrs to link text so {es}//{kib}
references in URL link text become {{es}}/{{kib}} instead of leaking
as single-brace literals
- ConvertCommand: conf.yaml chunk:N maps to chunkLevel N+1 in our AST
(chunk:1 → chunkLevel 2 creates separate pages for === sections,
matching elastic.co's page-per-section structure; 423 → 3376 pages)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…multi-line collection, title xrefs - Prevent double-backtick AsciiDoc curly-quotes from opening a code span by adding (?<!`) lookbehind on the mono pattern (group 16) - Restrict bold (groups 13,14) and italic (group 15) from matching across < so they cannot consume <<anchor>> xref openers in cross-mark content (e.g. *word*<<xref>>) - Fix ParseDescriptionList to collect consecutive Text tokens into one paragraph (matching ParseParagraph behaviour) so xrefs that wrap across continuation lines within a DL item are kept in a single ParseInlines call - Add SubstituteTitleXrefs helper to MarkdownEmitter and apply it to section headings, document titles, and callout descriptions, which previously used SubstituteTitleAttrs only and left <<anchor,text>> patterns unconverted - Add ifeval:: lexer fix: relax middle group from .+ to .* to allow empty attribute name - Add GetBlockAttributeRegex trailing-whitespace trim to avoid false negatives - Add test coverage: curly-quote context, ordered-list items, bold-asterisk interference Result: raw xref leftovers in en/elasticsearch/reference/8.19 drop from 273 to 9 files. The remaining 9 are source-level bugs (malformed single >), code-block placeholders, or xrefs genuinely inside code spans -- all correct not-converted behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a pipe table contains an include:: on its own line (typical
for shared common-options snippets), the lexer emits it as a Text
token (not IncludeDirective) because the inTable state flag is not
set during include resolution. ParseTable now detects the include::
pattern in Text tokens, reads the referenced file, and appends its
| ... | rows directly to the current table instead of leaking the
directive text into the last cell.
Also handles the IncludeDirective token type for future-proofing.
Result: include:: leftovers drop from 58 to 24 (remaining 24 are
{esql-specs} references to external test fixture files that are not
part of the repository and cannot be resolved).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rseCalloutList for orphan markers - After a listing block, skip any trailing comment tokens (// TEST[...]) and one optional blank line before attempting callout collection; restore position if no callout markers follow, preventing skipped blank lines from orphaning later content - Add ParseCalloutList routed from ParseBlock for standalone <n> callout lines that appear after continuation text or in other contexts where the code-block collector already broke; collects multi-line descriptions via continuation join - Lexer: relax ifeval:: middle group .+ → .* to allow empty attribute name - Lexer: allow trailing whitespace on block attribute lines [attr]\s* Result: raw <n> callout markers in en/elasticsearch/reference/8.19 drop from ~396 to ~90 files. Remaining cases are multi-callout blocks where <1> description wraps across lines, breaking the code-block collector before reaching <2>+. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ParseParagraph: stop collecting Text tokens at callout markers (<n>) so ParseCalloutList gets to handle the run instead of having them swallowed into paragraph content - ParseDescriptionList: same guard in the inline Text-continuation loop - Code-block callout collector: collect continuation lines for each <n> description so multi-line callout text is fully captured rather than truncated at the first line - ParseDescriptionList: when inline description text (after ::) contains an unclosed <<, join continuation Text tokens until the xref closes, enabling multi-line xref spanning within DL items Result: callout markers drop from 90 to 0; residual raw xrefs drop from 23 to 22 all of which are either inside code spans/fences (correct behaviour) or source-level malformed >> (not fixable). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pre-parse docs-repo/shared/attributes.asciidoc using the existing AsciidocParser
with branch/version seed attributes already set, so feature and product-name
abbreviations like {transform}, {ilm-init}, {anomaly-detect} and cross-guide
URL attributes like {ref}, {logstash-ref} are resolved at conversion time.
ProductNames keys (es, kib, etc.) are still excluded from eager expansion so they
continue to emit as {{name}} docs-builder substitution placeholders that get
resolved from the generated docset.yml subs: block.
Adds AsciidocParser.LoadAttributeFile static helper and ResolvedAttributes
property to support this pre-parse pattern.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add group 23 to InlineCombinedRegex to match ``text'' (AsciiDoc opening double-backtick + closing double-apostrophe) and emit as "text" (straight double-quote pair). The existing lookbehind on the mono group already prevents `` from opening a code span, so group 23 matches the remaining cases that fall through to raw text. Eliminates 21 occurrences of visible AsciiDoc typographic quote syntax in en/elasticsearch/reference/8.19. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…f section level - Add IsIncludeRoot flag to SectionNode to mark sections produced at the top level of a ProcessInclude call - Set IsIncludeRoot=true in ParseSection's IncludeDirective handler and in ProcessInclude's main loop - Fix chunkLevel calculation: conf.yaml chunk:N maps directly to AST level N, so pass book.Chunk directly instead of book.Chunk+1 - Fix ExtractPages to recurse into inline sections (level > chunkLevel) when looking for nested IsIncludeRoot sections — this handles the case where an include appears after a discrete section, which nests the included content under the discrete section in the AST - Fix CollectAnchors to also recurse into inline sections for the same reason - Add IncludeChain_EachIncludedFile_BecomesASeparatePage test that mirrors the elastic.co search-your-data structure with 3-level include nesting and discrete sections Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
docs-builder's anchor registry includes H2+ heading slugs and $$$...$$$ inline anchors but NOT H1 heading slugs. Cross-references like [text](page.md#page-anchor-id) were triggering 96 'does not exist' errors because the page-level anchor (from the original [[block-anchor]] before the first section) was emitted as '# Title [#anchor-id]' — a format docs-builder parses for H2+ but does not register in the anchor collection for H1. Fix: emit $$$anchor-id$$$ on a standalone line before H1 headings (both from document.Title and from Level=0 SectionNode). The $$$...$$$ syntax is the docs-builder InlineAnchor format, which is collected via document.Descendants<InlineAnchor>() and included in markdown.Anchors. With this fix: 0 errors and 0 warnings from docs-builder serve. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…gh bug - Extend ResolveVerbatimIncludes to handle standard include::[tag=] and full-file include::[] directives inside verbatim blocks (fixes 164 ESQL pages) - Strip ifeval::/ifdef::/endif:: conditional markers from verbatim block content where they can't be evaluated (fixes 2 files) - Add lookbehind to constrained +..+ passthrough regex so '8.0+' and similar version indicators don't start a passthrough span (fixes xref leaking past the passthrough into prose) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LF2iCfVK3ecJgPeFCP6p6S
…title Previously every included file became its own page (IsIncludeRoot), producing ~2065 flat pages with 27 duplicate slugs. The live guide at elastic.co uses asciidoctor's chunk_level rule: a section is a page iff it is not [discrete] and its level <= chunk + 1. This commit replicates that rule exactly. Changes: - PageChunker: rewrite as a single Traverse pass that returns a page tree instead of a flat list. IsPage = !IsDiscrete && (Level==0 ? IsIncludeRoot : Level <= chunkLevel+1). A single Level-0 child of document.Children is the book root and becomes index.md with its id mapped to "index". Duplicate slugs get _2/_3 suffixes with an onDiagnostic callback. - SectionNode: add IsDiscrete (from [discrete]/[float] block attribute). - AsciidocParser: pass pendingBlockAttr into ParseSection so [discrete] is recorded on SectionNode; keep IsIncludeRoot only for Level-0 distinction. - MarkdownEmitter: add HeadingLevelBase / UpdateHeadingBase so child headings are rebased relative to the page root (targz Level-2 page → ## Next steps). - YamlWriter: add Children to TocEntry; WriteTocYaml is now recursive. - ConvertCommand: WritePages recurses page.Children for nested toc entries; WriteBookVersionIndex emits navigation_title frontmatter. - PageChunker.ExtractDisplayTitle: handle ParagraphNode in addition to PassthroughNode for <titleabbrev>; emit navigation_title frontmatter when the abbreviated title differs from the section title. - Tests: update IncludeChain test to assert tree nesting; add ChunkerTests.cs covering discrete sections, deep-section inlining, navigation_title, auto-id, duplicate slug suffixing, and nested toc.yml indentation. Result for ES 8.19 (chunk:1): 2085 pages in a nested toc (was 2065 flat with 27 collisions), no duplicate toc entries, getting-started.md has navigation_title "Basics: Index and search using APIs", targz.md contains "## Next steps" instead of a separate next-steps.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When migration/index.asciidoc includes migration_intro (= Level-0 section) followed by individual migrate_8_N files (== Level-1 sections), the flat result list from ProcessInclude lacked the AsciiDoc-native parent-child relationship: L1 version pages appeared as top-level siblings of the L0 "Migration guide" page instead of its children. Add NestSectionsByLevel() called at the end of ProcessInclude: sections at level N following a section at level < N in the flat result list are re-nested as children of that ancestor, replicating what asciidoctor does naturally when parsing a single file. The fast path skips the re-nesting when all sections in the result list are at the same level. Result for ES 8.19: migrating-8.0 through migrating-8.19 now appear as children of breaking-changes.md in toc.yml, matching the live guide. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
theletterf
approved these changes
Aug 11, 2026
…-navigation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On Windows, Path.GetFullPath("/base/foo.adoc") returns "C:\base\foo.adoc".
Test FileReader lambdas keyed on "/base/foo.adoc" never matched, so all
includes were silently skipped and every chunker/parser test that relies
on include chains produced only an index page.
Normalize paths in ReadFile before calling FileReader: flip backslashes
and strip any drive-letter prefix so the contract is always Unix-style.
YamlWriter used AppendLine which emits \r\n on Windows; WriteTocYaml
tests asserting StartWith("toc:\n") failed. Replace with explicit '\n'.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
AppendLine emits \r\n on Windows. MarkdownEmitter.WriteLine and the navigation_title frontmatter block in EmitPageNodes both used it, so TitleAbbrev_EmitsNavigationTitleFrontmatter failed with \r\n vs \n. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #3802. Fixes the converted navigation so it matches the live
elastic.co/guidesidebar exactly.Root causes:
IsIncludeRoot-only rule), producing ~2065 flat pages with 27 slug collisions instead of the correct ~2085 hierarchically nested pages.TocEntryhad noChildrenandWriteTocYamlwas non-recursive, so parent/child structure was never written totoc.yml.migration_intro.asciidoc+ individualmigrate_8_N.asciidocfiles) were assembled flat inProcessInclude, losing AsciiDoc's natural level-based nesting.Fixes:
PageChunker.csTraversepass producing a tree ofPageOutput.IsPage = !IsDiscrete && (Level==0 ? IsIncludeRoot : Level <= chunkLevel+1). Book root detection (single L0 child →index.md). Duplicate slugs get_2/_3suffixes.SectionNode.csIsDiscrete(from[discrete]/[float]block attribute).AsciidocParser.cspendingBlockAttrintoParseSectionforIsDiscrete. AddedNestSectionsByLevel()at the end ofProcessInclude— sections at level N following a section at level < N in the flat result list are re-nested as children, replicating AsciiDoc's hierarchical nesting (fixes the Migration guide pattern).MarkdownEmitter.csHeadingLevelBase+UpdateHeadingBaseso child headings rebase relative to the page root.YamlWriter.csTocEntrygainsChildren;WriteTocYamlis now recursive.ConvertCommand.csWritePagesrecursespage.Children; emitsnavigation_titlefrontmatter.ChunkerTests.csnavigation_titlefrontmatter, cross-include level nesting, and nestedtoc.ymloutput.Results for ES 8.19 (chunk:1):
toc.yml(was 2065 flat with 27 collisions)breaking-changes.md(= Migration guide) hasmigrating-8.0–migrating-8.19as childrengetting-started.mdhasnavigation_title: "Basics: Index and search using APIs"quickstart.mdhas## Requirementsinline (discrete sections stay on-page)targz.mdcontains## Next stepsinline (Level 3 rebased under Level 2 page)Test plan
dotnet test tests/Elastic.LegacyDocs.Migration.Tests/— 44 tests pass./build.sh unit-test— 314 tests passdotnet run --project src/tooling/docs-migrate -- convert --majors 1 --minors 1 --book en/elasticsearch/referencegrep -o 'file: .*' toc.yml | sort | uniq -d→ empty (no duplicate entries)https://www.elastic.co/guide/en/elasticsearch/reference/8.19/index.html🤖 Generated with Claude Code