perf: speed up subtree preprocess for large multi-post Org files - #782
Open
HaoZeke wants to merge 9 commits into
Open
perf: speed up subtree preprocess for large multi-post Org files#782HaoZeke wants to merge 9 commits into
HaoZeke wants to merge 9 commits into
Conversation
Cross-subtree links are rewritten to dummy file paths in the pre-processed buffer. org-element-interpret-data then drops the file: type for relative paths, so re-parse treats them as fuzzy and Org 9.7 aborts with "Unable to resolve link: ...pre-processed.org::...". - Keep path and search-option separate on dummy file links - After interpret-data, force file: prefixes on dummy targets - Do not skip same-file org-id links during rewrite (only other files) - Handle fuzzy reparsed dummy paths in org-hugo-link - Re-enable the commented cross-post link tests in all-posts.org
Re-enable cross-post cases now export to relref shortcodes instead of the previous pre-processed.org resolve error note.
Current Org citeproc no longer emits the .csl-left-margin style block before csl-bib-body; main has been red on this golden drift. Align the expected Markdown with the export CI produces so the suite can proceed past citation-csl.org.
org-hugo-export-wim-to-md only updated org-id-locations when the table was empty. After a prior export wrote a non-empty ~/.emacs.d/.org-id-locations (test suite shares HOME across sequential emacs -batch runs), sibling files in the current directory were never scanned, so cross-file id links such as org-roam-file-A -> B aborted with "Unable to resolve link". Always rescan .org files in default-directory before export.
Org now emits —/– instead of numeric entities in captions, and drops the trailing space inside timestamp spans. Align expected Markdown with CI export so all-posts.org diffgolden can pass.
Un-comment and fill the Hyperlinks page: external vs same-post vs cross-post links, native Org forms that export to relref, the pre-processed.org resolve failure under Org 9.7+, and org-id tips for id: links. Escape example shortcodes for the doc Hugo build.
org-hugo--get-pre-processed-buffer dominated single-subtree export on large files (issue kaushalmodi#732): full AST re-serialization via org-element-interpret-data, per-link org-export--collect-tree-properties for id links, and rewriting every link in the file even when exporting one post. - Collect tree properties once for the whole preprocess pass - Copy the source buffer text and splice rewritten links in place instead of interpret-data on the full AST - For single-subtree export, only rewrite links inside that subtree; all-subtrees batch still rewrites every cross-post link once On test/site/content-org/all-posts.org (~9k lines), single-subtree export of "Links outside the same post" drops from ~3.2s to ~1.6s with no GC in the timed runs; golden output unchanged.
Buffer-copy preprocess preserves Org source timestamp spacing; interpret-data had normalized trailing spaces inside DEADLINE/SCHEDULED spans. Align the golden with the source-faithful export.
Single-subtree export no longer runs org-element-parse-buffer on the whole multi-post file. Build a destination index with one org-element-at-point per headline (cached on buffer-chars-modified-tick), parse only the exported subtree for links, and splice rewrites into a buffer copy. All-subtrees batch keeps the full-parse path for the shared pre-processed buffer. Benchmark on test all-posts.org (~9k lines), single-subtree export of "Links outside the same post": ~0.50s avg (was ~1.57s after buffer-copy, ~3.2s before). Golden Markdown unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses #732: single-subtree export on large multi-post Org files spent most of its time in
org-hugo--get-pre-processed-buffer(full-file parse, link rewrite, thenorg-element-interpret-datare-serializing the whole AST). That also drove the heavy GC in the profiler.Changes
org-export--collect-tree-propertiesonce per preprocess (was recomputed for everyid:link).org-element-interpret-dataon the full tree.file:link handling (org-hugo--ensure-preprocessed-file-link-types) and same-fileid:rewrite behavior from fix: restore same-file cross-post links under Org 9.7+ #781.Benchmark (local, Emacs 30.2)
File:
test/site/content-org/all-posts.org(~9100 lines). Export post Links outside the same post withorg-hugo-export-wim-to-md(3 timed runs after warm-up):Golden Markdown for that post is unchanged.
Test plan
make -j1 testmatrixNotes
Stacked on the Org 9.7+ cross-post correctness work (#781). Parse of the full buffer remains (~1.2 s on this file); further gains need a lighter destination index than
org-element-parse-buffer.