Skip to content

Commit 737f1d0

Browse files
dbdeveloperclaude
dbdeveloper
and
claude
committed
Merge branch 'sync2-worker-reorg' into main — 2.0.2-beta rework
46 commits across Stages 1-9 implementing a Web Worker orchestra, SHA-first reconcile, split Commit/Sync UX with cancellable sync, modify-in-place crash safety, and expired-token recovery. Architecture summary (full design rationale in docs/PSEUDO-MERGE-MODE.md §17-19; the per-stage plan that produced this branch was docs/tasks/SYNC2-WORKER-REORG.md, removed in this merge with leftovers preserved in docs/tasks/SYNC2-TODO.md): - Worker orchestra (Stages 3-6): CPU pool + dedicated network worker; main thread free during multi-MB CPU/network work; CORS-validated on Capacitor Android. - SHA-first reconcile (Stage 5): metadata-only fetches resolve ~75% of paths from SHAs alone, no blob fetch needed. - Settings rework (Stage 7): syncStartsWithCommit master toggle, showCommitRibbonButton, consolidateCommits, drain-status section with [Stop sync], TokenExpiredModal, Performance group with maxAutoMergeSizeBytes. - Crash safety (Stage 7): modify-in-place + .sync-tmp. marker preserves open editors without losing rename-strategy's forward recovery guarantee. - Perf instrumentation (Stage 8): CPU micro-benchmarks for diff3, base64, SHA, Worker dispatch under tests/perf/perf-cpu-*. Tests at merge time: 679 unit pass; 114 integration pass (last full run pre-merge). No breaking changes for users on the default configuration. Renamed data.json keys (autoCommitOnSync, accumulateOfflineSyncs) are still loaded transparently; first-load Notice instructs the user to update each device's data.json by hand. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 parents fead510 + b93eb72 commit 737f1d0

36 files changed

Lines changed: 4754 additions & 205 deletions

‎CHANGELOG.md‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,145 @@ For the full design rationale behind any change below, see
1111

1212
---
1313

14+
## 2.0.2-beta — 2026-05-30
15+
16+
Major architectural rework, no breaking changes for the default
17+
configuration. The engine now runs CPU- and network-heavy work in
18+
a small Web Worker orchestra so the UI stays responsive even
19+
during multi-megabyte syncs on Obsidian Mobile. New optional
20+
split between **Commit** (stage changes locally) and **Sync**
21+
(upload to GitHub) for users who want to batch commits offline
22+
and push only on WiFi. A new **Settings → GitHub sync status**
23+
section gives a live view of the current upload + the cancel
24+
button. Expired GitHub tokens now surface as a step-by-step modal
25+
instead of silent 5-minute retry loops.
26+
27+
If you're a single-device user with `Sync starts with commit:
28+
true` (the default) and `Show commit ribbon button: false` (also
29+
the default), nothing about your day-to-day workflow changes —
30+
the **Sync with GitHub** button still commits + pushes in one
31+
click. Everything below is additive.
32+
33+
### Added
34+
35+
- **Worker orchestra (Stages 3-6).** A small pool of Web Workers
36+
now handles 3-way text merge, SHA computation, base64 decode,
37+
and every GitHub API call. The main thread stays free for the
38+
editor; "click Sync, keep editing" works as a hard guarantee
39+
even at multi-megabyte file sizes. Threshold-gated so small
40+
operations stay inline (no postMessage overhead). The CORS
41+
feasibility was validated on Capacitor Android (Pixel 6 Pro) —
42+
worker `fetch` against `api.github.com` round-trips in
43+
~800 ms.
44+
- **SHA-first reconcile (Stage 5).** Reconcile now fetches GitHub
45+
metadata (sha + size) before pulling blob content. ~75 % of
46+
paths resolve from SHAs alone (no remote change / ours wins /
47+
already in sync) — no blob fetch, no base64 decode, no merge.
48+
Saves multi-MB of redundant downloads on every multi-device
49+
sync.
50+
- **Split Commit / Sync (Settings → "Sync starts with commit",
51+
default ON).** Turning the master toggle OFF means the
52+
**Sync** button only uploads what's already staged; committing
53+
becomes a separate **[Commit]** ribbon button (also a
54+
command-palette entry and a hotkey-bindable action). Useful
55+
for staging commits offline and pushing only on WiFi.
56+
- **Show commit ribbon button toggle (Settings, default OFF).**
57+
Independent of the master toggle. Lets the user surface a
58+
**[Commit]** ribbon icon even when Sync still commits + uploads
59+
in one click.
60+
- **"Upload pending commits to GitHub (no new commit)" command.**
61+
Pure-drain hotkey for users who hide all ribbon icons.
62+
Independent of the master toggle.
63+
- **Settings → GitHub sync status section.** Top of the page,
64+
always visible. Shows: live timer while a sync is running, the
65+
file being processed (N of M counter), last error (with
66+
elapsed time prefix), and a `[Stop sync]` button to cancel an
67+
in-flight upload.
68+
- **`Sync2Manager.cancelDrain()` API.** Cancellation propagates
69+
via an abort flag the reconcile loop checks between files.
70+
- **Maximum auto-merge file size (Settings → Performance, default
71+
1024 KB).** Exposes the previously-hardcoded
72+
`RECONCILE_AUTO_MERGE_LIMIT`; tune up if your corpus needs
73+
auto-merge on bigger files (validate on the slowest device
74+
first — see `tests/perf/README.md`).
75+
- **Expired-token modal.** When GitHub returns 401 ("Bad
76+
credentials") the plugin now opens a guided recovery modal:
77+
intro + the three steps + buttons to open the GitHub token
78+
page, the README walkthrough, and the plugin's own settings
79+
tab. Throttled to once per hour. Catches the common
80+
"fine-grained PAT hit its 366-day cap and the daily 5-minute
81+
interval drain silently fails" scenario.
82+
- **Modify-in-place crash safety (Stage 7).** When the engine
83+
writes to a file that's currently open in an editor, it now
84+
uses `vault.modifyBinary` to preserve the editor's cursor and
85+
scroll position — instead of the rename-aside strategy that
86+
closes the view. Backed by a marker-file protocol that
87+
forward-completes any interrupted modify on next plugin
88+
onload.
89+
- **CPU micro-benchmarks under `tests/perf/perf-cpu-*`.** Diff3
90+
timing matrix, base64 decode, git-blob SHA, WorkerClient
91+
dispatch overhead. Baselines documented in
92+
`tests/perf/README.md`.
93+
94+
### Changed
95+
96+
- **`Sync2Manager.timed()` HTTP wrapper routes through the
97+
network worker** when `WorkerClient` is provided. Existing
98+
`requestUrl` path stays as fallback for the Settings-tab
99+
connection probe and unit tests.
100+
- **Reconcile loop reads ours bytes FIRST**, then fetches base +
101+
theirs metadata. Together with the SHA-first decision tree
102+
this means the cheap local read happens before any network
103+
round-trip, and the expensive byte-fetches only happen for
104+
paths that truly need them.
105+
- **`setTimeout(0)` yields between reconcile path iterations.**
106+
Prevents the macrotask queue from starving during long drains.
107+
- **`PushQueue.readFile` uses `fetch(getResourcePath(...))`.**
108+
Documented part of Obsidian's `DataAdapter` API; on mobile
109+
this resolves to a `http://localhost/_capacitor_file_/...`
110+
URL that bypasses the JS↔native bridge. Faster and noticeably
111+
more reliable than `adapter.readBinary` for multi-MB files.
112+
113+
### Renamed
114+
115+
The following data.json keys were renamed. Existing values are
116+
still loaded transparently and a one-time log line + 30-second
117+
Notice on first run instructs you to copy the new shape
118+
manually. The OLD keys keep working (loaded but no longer read).
119+
120+
- `autoCommitOnSync` → `syncStartsWithCommit` (also unifies
121+
manual / interval / startup behaviour under one toggle —
122+
default `true` to preserve today's manual-click semantics)
123+
- `accumulateOfflineSyncs` → `consolidateCommits`
124+
- "Push plugins data.json to GitHub" (UI label) → "Sync plugins
125+
data.json"
126+
127+
### Removed
128+
129+
- The 9 mobile-diagnostic buttons under Settings → "Mobile
130+
diagnostics" (added during the May 2026 field investigation).
131+
The empirically-validated learnings landed as proper engine
132+
changes; the diagnostic surface is now redundant.
133+
134+
### Fixed
135+
136+
- **Open file no longer closes when a remote change lands.**
137+
Was caused by the atomic-rename strategy (renames the live
138+
file aside, then renames new bytes over the slot — Obsidian
139+
sees the file disappear and unloads the editor). Replaced with
140+
`vault.modifyBinary` for existing TFiles; the rename strategy
141+
still runs for brand-new files where there's no open editor
142+
to preserve.
143+
144+
### Internal
145+
146+
- 49 commits on `sync2-worker-reorg`, 51 if you count the merge
147+
commit. Branch cut from `main` at `fead510` (2.0.1-beta5).
148+
- 679 unit tests + 114 integration tests + 24 perf baselines.
149+
All green at merge.
150+
151+
---
152+
14153
## 2.0.1-beta5 — 2026-05-29
15154

16155
Hotfix release. Fixes a critical sync bug that could overwrite

‎CLAUDE.md‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -57,10 +57,18 @@ src/
5757
│ # 4 status subclasses, PlatformError, StaleStateError, makeGithubAPIError
5858
│ # dispatcher. PSEUDO-MERGE-MODE §13.
5959
├── github/client.ts # Thin requestUrl wrapper, retryUntil; throws via makeGithubAPIError;
60-
│ # getContentsAtRef does Blobs-API fallback for >1MB files (PSEUDO-MERGE-MODE §16.6)
60+
│ # getContentsAtRef does Blobs-API fallback for >1MB files (PSEUDO-MERGE-MODE §16.6);
61+
│ # every HTTP call routes through WorkerClient.httpRequest when one is wired
6162
├── settings/
62-
│ ├── settings.ts # GitHubSyncSettings + DEFAULT_SETTINGS
63-
│ └── tab.ts # Settings UI (trim onChange, Reset modal)
63+
│ ├── settings.ts # GitHubSyncSettings + DEFAULT_SETTINGS (syncStartsWithCommit,
64+
│ │ # showCommitRibbonButton, consolidateCommits, maxAutoMergeSizeBytes)
65+
│ └── tab.ts # Settings UI (trim onChange, Reset modal, GitHub sync status section,
66+
│ # Performance group with max-auto-merge KB input)
67+
├── worker/ # Web Worker orchestra (PSEUDO-MERGE-MODE §17). esbuild emits each entry
68+
│ ├── types.ts # point as an IIFE, inlines as string via `define`, runtime wraps in Blob URL.
69+
│ ├── cpu-worker.ts # CPU pool: decode-base64, compute-git-blob-sha, merge-text (bundles node-diff3)
70+
│ ├── network-worker.ts # Single dedicated thread; native fetch executor for every GitHub HTTP call
71+
│ └── worker-client.ts # Main-thread controller; pool dispatch, request-id multiplex, terminate, fallback
6472
└── sync2/
6573
├── sync2-manager.ts # Orchestrator: syncAll, syncFile, drain, processBatch,
6674
│ # validateDeletionsAgainstHead (pre-flight, §12.1),
@@ -77,7 +85,9 @@ src/
7785
│ # Unicode), encodePathForGithub, safeRename. PSEUDO-MERGE-MODE §11.
7886
├── gitignore-invariants.ts # Invariant .gitignore blocks; always-write enforce
7987
├── commit-message.ts # Hardcoded format* helpers; commitMessageForBatch
80-
├── atomic-write.ts # 5-step atomicWriteFile + stagingPathFor + AtomicWriteRecovery.sweep
88+
├── atomic-write.ts # 5-step atomicWriteFile + stagingPathFor + AtomicWriteRecovery.sweep;
89+
│ # fast-path uses vault.modifyBinary for open TFiles (preserves editor cursor/scroll)
90+
│ # via a .sync-tmp + .<basename>.sync-tmp. marker forward-recovery protocol
8191
├── conflict-store.ts # ConflictRecord + 3-step create + renameVaultSiblingsToUnresolved
8292
├── conflict-classifier.ts # Pure classify() + evaluateConflictState (Phase A + Phase B)
8393
├── conflict-watcher.ts # vault.on listener; READ-ONLY counter.markDirty()
@@ -90,7 +100,8 @@ src/
90100
├── types.ts # QueueBatch, FileChange, EnqueueMeta
91101
└── views/
92102
├── conflict-status-indicator.ts # Status-bar 🔀 count
93-
└── pre-sync-conflict-modal.ts # Pre-Sync confirmation modal
103+
├── pre-sync-conflict-modal.ts # Pre-Sync confirmation modal
104+
└── token-expired-modal.ts # 401 / 403 recovery dialog (Stage 7)
94105
```
95106

96107
## Testing
@@ -177,7 +188,11 @@ The bucket form takes a glob — `tests/integration/scenarios/sync2/conflicts*`
177188
- **Don't add files to the hardcoded `isSyncable` blocklist** without a real reason. The default for new "should we sync this?" rules is to add patterns to the seeded gitignore (`CONFIG_DIR_SEED` / `ROOT_SEED` in `gitignore-invariants.ts`) — that way users can opt out.
178189
- **Don't hand-edit the canonical block in `<configDir>/.gitignore`** — `GitignoreInvariants.enforce()` will rewrite it on the next plugin load. To customise the truly-required behaviour, edit the constants in `gitignore-invariants.ts` and ship a new build.
179190
- **Polling, not events, for the sync engine.** `findChanges` walks the vault on each sync click; no `vault.on` subscription for sync purposes. Implication: edits made while the plugin was disabled get picked up on the next sync click without any "missed events" failure mode. The conflict layer's `ConflictWatcher` IS event-driven (`vault.on('delete'|'modify'|'rename')`), but **read-only** — it only calls `counter.markDirty()`, never mutates store; all conflict mutations happen at drain-start. See [`docs/PSEUDO-MERGE-MODE.md`](./docs/PSEUDO-MERGE-MODE.md) §5.
180-
- **No scheduler logic in `main.ts`.** Periodic-tick decisions (interval enabled vs watchdog vs `autoCommitOnSync`) and the onload-startup pulse live in `src/sync2/interval-scheduler.ts` so they can be unit-tested in isolation under a fake timer. If you find yourself adding an `setInterval` or `app.workspace.onLayoutReady` callback for sync purposes inside `main.ts`, move it into `IntervalScheduler` instead.
191+
- **No scheduler logic in `main.ts`.** Periodic-tick decisions (interval enabled vs watchdog vs `syncStartsWithCommit`) and the onload-startup pulse live in `src/sync2/interval-scheduler.ts` so they can be unit-tested in isolation under a fake timer. If you find yourself adding an `setInterval` or `app.workspace.onLayoutReady` callback for sync purposes inside `main.ts`, move it into `IntervalScheduler` instead.
192+
- **Worker orchestra: CPU pool + dedicated network worker.** Stage 4-6 of the 2.0.2-beta rework moved every hot-path CPU operation (3-way merge, base64 decode, SHA computation) and every GitHub HTTP call off the main thread. The orchestra lives in `src/worker/`; esbuild emits each worker entry point as a standalone IIFE and inlines the source as a string constant via `define`, so `main.js` ships a single bundle. Runtime wraps each string in a `Blob` URL and constructs `new Worker(url)` from it — no `importScripts`, no separate file fetch, no Capacitor `app://` URL ambiguity. Workers CANNOT touch any Obsidian API (`vault.adapter.*`, `app.workspace`, settings) — those stay on main. **All HTTP calls from the engine MUST go through `WorkerClient.httpRequest`** (CORS-validated against `api.github.com` on Capacitor Android). The Settings-tab connection probe is the one allowed exception — it uses `requestUrl` directly so a click never touches plugin state.
193+
- **Modify-in-place uses `vault.modifyBinary` + a `.sync-tmp.` marker for crash safety.** When the engine writes to a file that already exists as a TFile, `atomicWriteFile` takes a fast path that preserves any open editor's cursor + scroll position. Protocol: stage new bytes in `<file>.sync-tmp.<ext>` → drop a zero-byte marker at `.<basename>.sync-tmp.` (leading + trailing dot — syntactically distinct from staging files) → `modifyBinary(target, newBytes)` → cleanup. On crash, `AtomicWriteRecovery.sweep` sees the marker, renames sync-tmp over the target (forward-complete), and removes the marker. Recovery runs at plugin onload BEFORE `workspace.onLayoutReady` so the rename's editor-close side effect is moot. The rename strategy still runs for brand-new files (no existing TFile to modify); SHA-based recovery handles its `.sync-bak` orphans, unchanged from 2.0.1.
194+
- **`syncStartsWithCommit` master toggle controls all sync surfaces (default `true`).** Manual `[Sync]` click, interval tick, and startup sync all branch on this single setting. `true` → commit + drain (today's manual-click semantic; preserves backward compat). `false` → drain only; commit becomes the user's separate action via the `[Commit]` ribbon button or the `commit-local` command. The `showCommitRibbonButton` toggle controls the ribbon icon independently — it's a UI affordance, not a semantic.
195+
- **`atomicWriteFile` is invoked from many places. Settings-tab UI text should NOT name engine concepts ("drain", "queue", "batch") — use plain English for users.** Engine identifiers (cancelDrain, DrainStatus, setDrainStatusListener) stay as code-level jargon because they're API names, not user copy. Stage 7 specifically swapped UI copy: "Drain status" → "GitHub sync status", "Stop drain" → "Stop sync", "Drain running" → "Syncing with GitHub".
181196
- **`drain()` is re-entrant-safe via a `running` flag** on `Sync2Manager`. Concurrent `syncAll()` calls (e.g. interval tick fires while user click is mid-flight) collapse into one drain — the second call returns immediately. Don't bypass this with a separate code path; the integration suite's H3 test pins the serialisation.
182197
- **Commit messages are hardcoded** in `src/sync2/commit-message.ts` (`formatSyncMessage`, `formatResolveConflictMessage`, etc.). Don't reintroduce a per-user template field — the design choice was deliberate (date/time live in commit metadata; provenance lives in the trailing `(deviceLabel)` suffix).
183198
- **When working on conflict resolution OR the push pipeline OR cross-cutting infrastructure** (cross-platform contracts in `cross-platform.ts`, typed errors in `errors.ts`, pending-deletions in `pending-deletions-store.ts`, skip-class annotations in any loop), [`docs/PSEUDO-MERGE-MODE.md`](./docs/PSEUDO-MERGE-MODE.md) is the canonical spec the code targets. Code comments reference the article's section numbers (e.g. `§9.4`, `§10 Scenario E`, `§11 cross-platform contracts`, `§12.1 pre-flight validation`, `§13 error taxonomy`, `§14 skip-class`); use those to navigate between code and design rationale. The bug catalog in `§16 Field Postmortems` is the triage index for similar future symptoms.

‎README.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,11 @@ Version `2.0.1-beta` · AGPL-3.0 · Fork of
1111
## What's new
1212

1313
Per-release notes live in [`CHANGELOG.md`](./CHANGELOG.md).
14-
Latest: **2.0.1-beta5** — hotfix for >1 MB-file sync; see
15-
[CHANGELOG.md](./CHANGELOG.md#201-beta5--2026-05-29) for the
16-
recovery checklist if you were affected.
14+
Latest: **2.0.2-beta** — Worker orchestra, SHA-first reconcile,
15+
split `[Commit]` / `[Sync]` UX, cancellable in-flight uploads,
16+
expired-token modal. See
17+
[CHANGELOG.md](./CHANGELOG.md#202-beta--2026-05-30) for the
18+
full list.
1719

1820
The full design rationale behind any release is in
1921
[`docs/PSEUDO-MERGE-MODE.md`](./docs/PSEUDO-MERGE-MODE.md).
@@ -133,6 +135,8 @@ want your notes public → leave "Initialize with README" unchecked
133135

134136
<!-- SCREENSHOT: empty repo creation form -->
135137

138+
<a id="github-token-setup"></a>
139+
136140
#### 3. Generate a fine-grained Personal Access Token
137141

138142
[github.com/settings/personal-access-tokens/new](https://github.com/settings/personal-access-tokens/new)

0 commit comments

Comments
 (0)