Skip to content

Width ablation at full-MP scale (W96/W128/W160) + CoreWeave GB200 training infra - #160

Draft
forklady42 wants to merge 59 commits into
mainfrom
betsy/coreweave-w96
Draft

Width ablation at full-MP scale (W96/W128/W160) + CoreWeave GB200 training infra#160
forklady42 wants to merge 59 commits into
mainfrom
betsy/coreweave-w96

Conversation

@forklady42

@forklady42 forklady42 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Width ablation of the RHOAR-Net ResUNet3D at full-MP scale (capped GGA+GGA+U, 111,257 structures), run as a single-variable ladder — each config differs from its neighbor only in model.n_channels — on CoreWeave GB200 nodes via Marin's Iris scheduler. This PR carries the configs, the CoreWeave training infrastructure, and the LR/WD sweep work.

Headline: W128 reaches 0.481% combined NMAE on the withheld test set (GGA 0.480% / GGA+U 0.483%) — below the 0.5% ChargE3Net-threshold target. Validation curves (as of 2026-08-11, W128/W160 still training):

width params epochs best val NMAE status
W64 (ref) 48M 16 0.840% June campaign, 8×A100 (fp16→bf16) — not epoch/hardware-matched
W96 108.8M 28 0.685% complete
W128 193.4M 74+ 0.561% training; test-set 0.481% (ep-69 ckpt, job_w128_test_eval.slurm)
W160 302.1M 39+ 0.583% training; ~4% ahead of W128 at matched epoch

Same-age ordering is strictly monotone in width at every epoch. W96→W128→W160 ran with identical data, splits, and hyperparameters on identical hardware (4×GB200 each).

Configs (the ablation chain)

  • config_gga_gga+u_w96.yaml / _w128.yaml / _w160.yaml — the single-delta ladder; headers document memory envelopes, comparability rules (do not re-cap the filelists), and the CoreWeave staging paths.
  • config_gga_gga+u_w192.yaml / _w256.yaml — exploratory, blocked: cuDNN's fast conv engines fall back to reference kernels (~400× slower) when the decoder-concat tensor exceeds ~2e9 elements. Measured ceilings on the unmodified capped dataset: W160 safe (2.4 s cap-grid steps), W176+ not (590 s). Headers carry the full analysis; scripts/coreweave/count_voxel_bands.py quantifies width-specific cap costs (W192 would drop 0.78% of data, W256 4.3%).
  • config_gga_gga+u_w128_lr2e-3.yaml — LR ablation at W128: null result (converges onto the base curve by epoch ~19; halted).
  • config_gga_gga+u_w128_test_della.yaml + job_w128_test_eval.slurm — withheld-test-set evaluation (Della).
  • LR/WD-vs-width sweep (stages A–C): configs + docs/lr_wd_width_sweep.md.

CoreWeave/Iris training infrastructure (scripts/coreweave/)

  • stage_data.sh — CAIOS→node-NVMe staging via rclone/LOTA (~8.5 min cold for 1.22 TiB / 678K objects), idempotent marker, RhoRead symlink shim. CAIOS requires virtual-host addressing (s5cmd is path-style-only and cannot be used).
  • run_training.sh — stage → checkpoint restore → bf16 torchrun (single node, 4×GPU) with wandb-offline sidecar sync and 10-min checkpoint sync to object storage. Resume logic is bucket-authoritative and hardened by three production incidents: always reconcile last*.ckpt with --update, promote the newest, and refuse to start if the promoted checkpoint is >10 min staler than the bucket's (stale node copies twice caused multi-epoch rollbacks under heavy preemption churn).
  • submit.sh — one-command laptop-side Iris submission (batch priority, 25 preemption-retries; jobs survived 30+ preemptions/day during Marin campaign contention with ~15–25 min loss per bounce).
  • dry_run.py — pre-launch validation: dataloader path, memory envelope at the 180³ grid cap, grid-ladder/cudnn-benchmark probe flags (used to map the kernel cliff).
  • train.py changes: cu128 wheels for linux-aarch64 (PyPI ARM torch wheels are CPU-only), step-based interval checkpointing (the wall-clock trigger's DDP broadcast deadlocked a 4-rank run), wandb runs named w{width}_{MMDD-HHMM}, wandb offline mode (online init crashes with the viewer flags=null client bug and a rank-0 death silently deadlocks DDP).

Notes & caveats

  • W64 is a directional reference only (different hardware/precision history); the controlled comparison is W96/W128/W160.
  • Production MFU ≈ 9–13% (batch-1 variable-grid 3D convs are occupancy-bound; measured 23.7 MFLOPs/voxel fwd at W96). Levers documented for future work: grid-size bucketing, torch.compile, channels-last-3d.
  • Checkpoints: every epoch best in s3://rhoarnet-us-east-08a/checkpoints/, campaign bests pinned under checkpoints/archive/ and mirrored to Della scratch. W&B: PrinceOA/mp-gga-ggau-width.
  • Draft while W128/W160 finish; final numbers and the val-curve figure will be added at halt.

🤖 Generated with Claude Code

forklady42 and others added 30 commits May 28, 2026 11:02
@forklady42 @claude
Add large-scale GGA + GGA+U fp32 training config
b79d6cb 
Copied from della experiment dir for the large_scale/gga_gga+u_f32 run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Add Modal training + Globus data-load for large-scale GGA/GGA+U run
2e2ce78 
Bring PR #100's modal/ toolkit into this branch and adapt train.py to be
config-driven and multi-dataset aware: remap della share paths onto the
electrai-data Volume, verify per-dataset filelists, namespace checkpoints by
run_name for auto-resume across 24h-capped runs, and default to a single A100.

Add modal/globus_load.py to host Globus Connect Personal inside a Modal
container (Volume mounted, non-interactive setup-key) for direct della->Volume
transfer, and a subset/2-epoch smoke config for first-run validation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Add prep_volume helper to wire the data Volume after Globus transfer
0a1e0f6 
prep_volume (re)creates the functionals/{gga,gga+u}/{data,label} symlinks into
the transferred rho_* dirs, builds the subset smoke filelists, sanity-checks the
first id resolves to a real .zarr, and commits the Volume explicitly. Point the
smoke config comment at it instead of manual modal shell steps.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Point populate_volume at oa-electrai with dedicated read-only secret
c841b35 
Swap the Modal secret from aws-credentials (elvis-modal-etl, no oa-electrai
access) to oa-electrai-read (dedicated electrai-modal-reader IAM user,
least-privilege read on mp/chg_datasets/*). Update defaults to the new
bucket/prefix/dest and bump the function timeout to 24h for the full ~1.25 TB
pull.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Parallelize populate_volume S3 download with ThreadPoolExecutor
35160da 
Serial boto3.download_file across ~680K files would take 12+ h; with 32
workers and a batched, periodically-committed loop it should land in
under an hour. Per-file errors are now logged and counted rather than
fatal, and the Volume is committed every 5000 completions so a preemption
loses at most one batch (already-downloaded files skip on resume via the
size check).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Use .spawn() so the populate_volume pull survives local CLI disconnect
f279059 
`.remote()` ties the remote function's lifetime to the local caller; in detached
mode Modal still cancels it if the caller disconnects (as a DNS blip
demonstrated). `.spawn()` is true fire-and-forget. Local entrypoint now spawns
the FunctionCall, logs its id + monitoring URL, and exits.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Pack zarr stores as .zarr.zip to fit under Modal Volume's inode cap
cf2e7fd 
Modal Volumes are limited to 500K inodes; the unpacked zarr v3 layout uses
~8 inodes per <id>.zarr/ store (3 files + 5 dir entries) which puts our
113K-sample set at ~1.8M inodes -- way over the cap (the previous pull
ENOSPC'd at 73% errors after hitting the limit).

populate_volume groups S3 keys by their .zarr/ store, downloads each
store's 3 inner files in memory, and writes them to a single <id>.zarr.zip
on the Volume (atomic via tmp+rename); standalone keys (filelists,
split files) pass through unchanged. Adds a --wipe-first flag for the
first packed re-run.

Loader (load_zarr) auto-detects the packed form via zarr.storage.ZipStore;
RhoData's format detection now accepts either .zarr.zip or .zarr; and
prep_volume's sanity check accepts both.

Net: ~226K + standalone files ~ 230K inodes -- well under the 500K cap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Use .spawn() in train.py so multi-day runs survive local CLI disconnects
1b65484 
Same pattern as populate_volume: long-running Modal jobs need both
`modal run --detach` and `.spawn()` (not `.remote()`) for true fire-and-forget;
without the spawn, a transient DNS or network blip on the local caller can
cancel the remote function. The smoke run survived because it was <20 min,
but the full ~2-week training campaign would almost certainly hit a blip.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Add Lambda Cloud runbook (scripts/lambda/) for the long-run training …
bee89bc 
…campaign

Modal is built for short serverless functions; for a multi-week training
campaign the 24h function timeout + capacity re-queueing + Volume inode cap
(which forced .zarr.zip packing) all add up. Lambda Cloud is a better
operational fit -- single reserved VM, local NVMe for data, no re-queueing.

This adds four idempotent shell scripts mirroring the Modal pipeline:
- setup.sh        : install uv/aws/tmux + uv sync + cred check
- data_sync.sh    : aws s3 sync packed zarr from oa-electrai to local NVMe
- prep_data.sh    : relink functionals/*/{data,label} + smoke filelists
- run_training.sh : launch training under tmux with auto-resume + S3 ckpt backup

Reuses the existing .zarr.zip loader, configs, and training entrypoint
unchanged; the runtime config is rewritten in place to remap della paths
to the local data root.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Source ~/.bashrc in lambda run_training.sh for non-interactive callers
fea5263 
Non-interactive bash shells (e.g. ssh remote-cmd, tmux new-session 'cmd')
skip ~/.bashrc, so WANDB_API_KEY exports there don't reach the training
process. Source it explicitly when the env var isn't already set.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Lambda: prep_data accepts both .zarr.zip and .zarr/ (S3 is unpacked)
6662ed6 
S3 holds the original unpacked zarr layout (~678K files: ~226K stores
times ~3 inner files + metadata). We packed into .zarr.zip during the
Modal populate_volume run, but never wrote those back to S3, so Lambda
syncs the unpacked form. RhoData/load_zarr already auto-detect either
shape via ZipStore vs DirectoryStore -- prep_data just needed the same
flexibility in its sanity check. Tightened data_sync's expected file
count to ~670K to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Lambda run_training: grep WANDB_API_KEY from ~/.bashrc, not source it
42ef893 
Ubuntu's stock ~/.bashrc starts with an early return for non-interactive
shells (`case $- in *i*) ;; *) return;; esac`), so plain `source` from a
script silently does nothing. Grep+eval the export line directly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Lambda: default data + checkpoints to NFS filesystem; fix run_training
1924e0e 
Three corrections after first smoke attempt on Lambda:
1. Default DATA_ROOT/CKPT_ROOT to /lambda/nfs/betsy-rhoarnet-hero/{data,
   checkpoints} so the data survives instance termination (the Lambda
   filesystem is mounted persistently and 5.9 PB capacity).
2. run_training: set -o pipefail so the actual uv exit code is captured
   through `| tee`. Without it, a missing uv looked like rc=0 success.
3. run_training: invoke uv by absolute path (UV_BIN, defaults to
   ~/.local/bin/uv) so tmux's non-interactive shell doesn't need to source
   the bashrc that adds it to PATH.
4. run_training: also remap /data/... (Modal-style paths in the smoke
   config) to the local DATA_ROOT in addition to della paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Lambda run_training: simplify /data/ -> DATA_ROOT sed (YAML list-dash…
3ffae7f 
… safe)

Previous regex anchored at start-of-line whitespace then root:|split_file:,
which misses YAML list items like `  - root: /data/...`. Just match the
unique path segment /data/mp/chg_datasets/ instead -- it's not ambiguous
with anything else in the configs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Lambda: auto-detect NFS_ROOT from /lambda/nfs/*
b4e16e0 
The persistent filesystem mount path is instance-specific (e.g. the second
instance landed on /lambda/nfs/betsy-rhoarnet-hero-2). Auto-detect by
listing /lambda/nfs/* rather than hard-coding the first instance's name.
Explicit NFS_ROOT= env var still overrides.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Lambda run_training: WANDB_MODE_OVERRIDE env var for offline/disabled…
c094da0 
… modes

Workaround for a wandb-server-side issue where the PrinceOA viewer query
returns `flags: null` and crashes wandb 0.23-0.27 clients at login. Until
that's fixed (server-side or by patched client), `WANDB_MODE_OVERRIDE=offline`
lets training skip the broken login path while still logging locally;
`disabled` skips wandb entirely.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Lambda run_training: anchor path-rewrite sed on leading space
fafb857 
The /data/ rewrite was matching paths that already had /data/ as a
substring after the della->DATA_ROOT rewrite produced
/home/ubuntu/data/..., causing /home/ubuntu/home/ubuntu/data/... in the
full config. Anchor on a leading space (only YAML-value paths qualify)
to make both substitutions independent.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Lambda multinode: drop-in 2-node DDP launcher + runbook
81c6ea0 
Strictly additive to the single-node Lambda runbook. Three new files under
scripts/lambda/, none of the existing scripts or configs are touched so the
in-flight 4 GPU run is undisturbed.

- run_training_multinode.sh: torchrun-based launcher. Same tmux session
  structure and path-rewrite sed as run_training.sh; auto-detects
  NCCL_SOCKET_IFNAME via `ip route get $MASTER_ADDR`; head node also
  starts the ckpt-backup-to-S3 and wandb-sync windows.
- MULTINODE.md: operator runbook covering preflight (private IPs, NCCL
  iface, port reachability), the allreduce smoke, switching from the
  in-flight 4 GPU run, LR scaling (flagged as a 1-epoch verification),
  and a multi-node failure-mode checklist.
- nccl_test.py: 50-line cross-node NCCL allreduce smoke; validates the
  fabric is healthy before touching the real training command.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@forklady42 @claude
Add PORT_PLAN.md: migration plan from H100:4 -> 2-node H100:16
de3edf7 
Companion to MULTINODE.md. MULTINODE.md is the static operator runbook;
PORT_PLAN.md is the migration plan for porting a *running* 4x campaign
to 2-node 16-GPU with the lowest risk and the earliest abort signal.

Adds explicit decision gates (smoke gate: >=3.5x current throughput, LR
gate: val_loss within 50% of 4x at the same epoch index), 7-phase
timeline (~12h end-to-end), and a risk register ranked by likelihood x
impact. Captures the LR-scaling open question (suggested starting point
2.5x current lr), the wandb run-id discontinuity (new offline-run-*
dir on the new head), and operational concerns specific to porting a
live run (when to cut over vs let the current epoch finish).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Add MONITOR.md + monitor_status.sh: monitoring script + LLM-driven ho…
5436fcd 
…urly cron template

Captures the monitoring pattern we've been running by hand for the
current H100:4 campaign:

- monitor_status.sh: ad-hoc one-shot status snapshot. Auto-detects NFS
  root, prints tmux session, mtimes, GPU util, current step from live
  pane, checkpoints, backup loop, wandb-sync loop.
- MONITOR.md: documents the 3-part liveness rule (train.log mtime,
  GPU util, last.ckpt mtime), the false-alarm patterns we hit (tail-of-
  log artifacts, transient stat ENOENT on NFS, 10-min-cadence loops
  looking stuck against arbitrary timestamps), the verbatim cron prompt
  template for re-creating the hourly LLM-driven check, and a table
  of intervention thresholds.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Lambda monitor: EC2 resident Claude agent (active operator)
483b306 
Adds an always-on EC2 monitor that reuses the MONITOR.md liveness rule and
monitor_status.sh snapshot, but runs a resident Claude loop that auto-remediates
a bounded set of failures (whole-session / dead-window / dead-worker restarts,
NCCL-hang relaunch) and escalates everything else to Slack.

- monitor_agent_prompt.md  operator prompt (recall->probe->judge->act)
- MONITOR_EC2.md           runbook: allowed vs escalate + circuit breaker, provisioning
- monitor_loop.sh          resident supervisor; journal-as-memory, heartbeat on success
- monitor_status_all.sh    multi-node wrapper over monitor_status.sh
- monitor_settings.json    fail-closed permission allowlist
- monitor_watchdog.sh      independent who-watches-the-watcher
- monitor.env.example      targets + secrets template
- systemd/                 service + watchdog timer

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Lambda monitor: default to Opus (claude-opus-4-8) for monitoring
af2b108 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Lambda monitor: escalation degrades to a journal banner when no Slack…
f1aa848 
… webhook

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Lambda monitor: fix snapshot cmd allowlist match; wire Slack state vi…
8f44ba0 
…a CONTEXT

Snapshot command no longer uses an env-var prefix (HOSTS/RUN are already
exported into the agent env), so it matches the Bash(bash scripts/...) allow
rule instead of being denied every tick. Escalation curl is now a plain,
allowlisted command gated on a CONTEXT 'Slack escalation channel' flag.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Add width-64 GGA+GGA+U config + w64 mode for Lambda training
8a34d49 
New config_gga_gga+u_w64.yaml mirrors the mp-large-scale full run
(config_gga_gga+u_f32.yaml: full 113K GGA+GGA+U data, 100 epochs,
lr 0.001) with three deltas: n_channels 32->64, precision 32->16-mixed
(W64 OOMs at fp32 on variable-grid MP data), warmup_length 1->4.
Logged to W&B project mp-gga-ggau-width.

run_training.sh gains a `w64` mode pointing at the new config, and the
final wandb URL is now derived from the config instead of hardcoded to
mp-large-scale.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@forklady42 @claude
W64 run: grid-size cap + DDP timeout + frequent resume checkpoint
7c7fc95 
The W64 fp16 full-MP run crash-looped: the largest structures (grids up to
~540^3 vs a ~109^3 median) make a single DDP step exceed the 30-min NCCL
watchdog timeout, aborting all ranks. With no checkpoint before epoch end,
each restart began the epoch from scratch.

Three fixes:
- scripts/lambda/cap_filelists.py: generate *_capped filelists/splits dropping
  structures >5,832,000 voxels (~180^3, ~1.5% of data); remaps the positional
  split.json indices. config_gga_gga+u_w64.yaml now points at the capped files.
- train.py: raise DDP/NCCL collective timeout 30min -> 2h (safety net for the
  occasional slow sample).
- train.py: add a monitor-less ModelCheckpoint saving last.ckpt every 20 min
  (wall-clock) so a crash resumes mid-epoch instead of restarting it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@forklady42 @claude
W64: switch precision fp16-mixed -> bf16-mixed (fp16 diverged to NaN)
3c70889 
The 8x A100 resume ran clean through epoch 2 (val_loss 0.087->0.020->0.014)
then train_loss_step jumped to NaN mid-epoch-3 while the LR was still ramping
through warmup — an fp16 activation overflow (single-step 0.0189->nan, not a
gradual divergence). bf16-mixed has fp32's exponent range and the same memory
footprint, so it avoids the overflow. Resuming from ckpt_epoch=02.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Add train_watchdog.sh: cron-driven NaN/stall/down alert + auto-stop
2f79873 
Catches the failure that silently burned ~2 days on 2026-07-13 (training went
NaN but kept "running"; save_top_k masked it). One-shot, run from cron every
~10 min. On sustained NaN (2 strikes) it stops the run to halt compute waste;
also alerts on train.log stall and a missing tmux session. Writes ALERT.txt
(mirrored to S3 by the backup loop) + Slack if SLACK_WEBHOOK_URL is set.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@forklady42 @claude
Use PyTorch cu128 index for CUDA torch wheels on linux-aarch64
bb249dc 
PyPI's aarch64 torch wheels are CPU-only builds, so uv sync on Grace/GB200
nodes silently installs a torch that cannot see the GPU. Route torch and
torchvision through the PyTorch cu128 index (SBSA CUDA builds) for
linux-aarch64 only; x86 and macOS resolution is unchanged. Also refreshes
the stale uv.lock (torch 2.9.1 -> 2.10.0, matching the pyproject pin).

Verified on a CoreWeave GB200 node (sm_100): torch 2.10.0+cu128 sees the
GPU and runs bf16 conv3d forward/backward.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Add width-96 config for the width ablation (single-variable vs W64)
fe8342b 
config_gga_gga+u_w96.yaml is identical to config_gga_gga+u_w64.yaml (same
capped GGA+GGA+U dataset/splits, bf16-mixed, lr 0.001, warmup 4) except
n_channels 64 -> 96, so the only variable vs the W64 run is width. Targets a
GB200 (192 GB/GPU) so it fits without activation checkpointing, matching W64.
Compare against W64's val NMAE 0.84% (ckpt_epoch=16, run zz3oecp7) on this
exact dataset.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
forklady42 and others added 29 commits July 17, 2026 18:48
@forklady42 @claude
Add CoreWeave stage-in script for charge-density data
c6e9df1 
Syncs the dataset from the rhoarnet-us-east-08a CAIOS bucket to node-local
NVMe via the in-cluster LOTA endpoint, as the first step of an Iris task.
Idempotent via a completion marker so preemption retries on a warm node
skip the download.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42
Merge branch 'betsy/gga-ggau-w96' into betsy/coreweave-w96
8afaa9f
@forklady42 @claude
Point W96 config at the CoreWeave stage-in layout
7e8b068 
Data roots, split files, and ckpt_path move from Della/Lambda paths to the
node-local NVMe root populated by scripts/coreweave/stage_data.sh.
Hyperparameters untouched: the run stays a single-variable width comparison
against the W64 result (zz3oecp7).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Add CoreWeave training wrapper and dry-run script
d7d4c69 
run_training.sh: stage data, restore last.ckpt from CAIOS on cold nodes,
background checkpoint sync every 10 min plus a final sync on exit, then
single-node torchrun. dry_run.py validates the staged data path through
RhoRead and measures the W96 bf16 memory envelope at the 180^3 grid cap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Switch staging and checkpoint sync from s5cmd to rclone
59051b0 
CAIOS rejects path-style list requests (PathStyleRequestNotAllowed) and
s5cmd cannot emit virtual-host addressing against a custom endpoint;
rclone with force_path_style=false is already proven against this bucket.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Stage into /uv/cache: the only host-persistent mount in Iris task pods
e04fc55 
/mnt/local/iris-cache itself is not mounted into task pods (only its
uv-cache and cargo subdirs are), so the previous destination wrote to the
container overlay and was SIGKILLed at the 60GB ephemeral-storage limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Run wandb offline with a sync sidecar on CoreWeave
4b85565 
Online wandb.init crashes with the viewer flags=null TypeError (same client
bug that hit lambda2) and the resulting rank-0 death deadlocks DDP with the
other ranks spinning in the NCCL barrier. Offline init never contacts the
server; the wrapper re-syncs offline run dirs on the checkpoint interval
and once more at exit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Use step-based interval for the resume checkpoint, not wall-clock
616991a 
train_time_interval requires a DDP broadcast to align ranks on whether the
interval elapsed, and per-rank clocks can disagree: a 4-rank GB200 run
deadlocked mid-epoch with one rank in the ModelCheckpoint broadcast and the
others spinning in a different collective (NCCL watchdog never fired,
py-spy confirmed). every_n_train_steps=2000 (~20 min at observed pace) is
identical on every rank, so no alignment collective is needed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Promote the newest last*.ckpt before resume
179935a 
Lightning versions the save_last file (last-v1.ckpt, ...) when an earlier
run's last.ckpt exists, while resume always reads last.ckpt — after one
restart, retries would silently resume from the stale pre-restart state.
Fetch all last*.ckpt from the bucket and promote the newest.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Add W128 config and generic Iris submit helper
8db696e 
config_gga_gga+u_w128.yaml: single delta vs W96 (n_channels 96 -> 128) so
width stays the only variable. run_training.sh now derives its checkpoint
paths from the config filename stem so a new config can never resume a
different run's incompatible last.ckpt. submit.sh wraps the laptop-side
Iris submission with local credential passing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Add exploratory W256 config for dry-run evaluation
6966ae8 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
dry_run.py: grid ladder, multi-iter timing, cudnn-benchmark probe flags
40e4a4f 
For diagnosing the W256 large-grid kernel cliff: 180^3 steps take ~1495s
(vs ~4s expected) under default cuDNN heuristics while 128^3 is normal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Add W192 config: next width rung if it clears the cuDNN kernel cliff
5d3bb7c 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Add voxel-band counter for width-specific kernel-cliff caps
333c1a2 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
dry_run.py: add --n-channels override for width-ceiling probes
37ff4a0 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Add W160 config: widest cliff-safe rung on the unmodified capped dataset
63325c0 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
submit.sh: raise default retry budget for preemption-heavy periods
fb6f51c 
Kueue preemption by higher-priority Marin campaigns lands as task
failures (dataloader SIGTERM), and a 3-retry budget orphans a run after
three bounces. Resume-from-bucket makes retries cheap, so default to 25.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Name wandb runs from cfg.run_name
07c80d7 
Auto-generated names made restart segments of different-width runs
indistinguishable in the project view; run_name encodes the width.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Shorten wandb run names to w{width}_{MMDD-HHMM}
1222a42 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Add LR/weight-decay-vs-width sweep: stage-A configs, split subsampler…
bbb924b 
…, AdamW option

Design in docs/lr_wd_width_sweep.md: 2-stage proxy sweep (12K train
subset, production val set, 8-epoch compressed cosine) at W32/64/96 to
de-confound the width ladder's shared lr=1e-3/wd=0 recipe, then
extrapolate a tuned recipe to W128/W160.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Sweep stage A: W64 lr=8e-3 edge extension + W&B review script
a91a49f 
W64's best (4e-3, val 1.592%) landed on the grid's top edge, triggering
the design's extend-one-point rule. Review script merges preemption
segments per (width, lr, wd) and works around the wandb flags=null
viewer bug.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Sweep stage B: WD grid at per-width lr* (32=2e-3, 64=4e-3, 96=2e-3)
0a678da 
W64 tie-break: 8e-3 edged 4e-3 on the proxy (1.580 vs 1.592%) but
within noise and after near-divergence at ep2, so the design's
near-tie-goes-lower rule picks 4e-3.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
review_lrwd_sweep: keep _rep2 noise-bar reruns separate from stage-A …
b10cf57 
…twins

They share (width, lr, wd), so keying on the config run_name is the only
thing that stops the aggregator from silently merging the comparison the
reruns exist to make.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Sweep doc: record stage A+B results and stage-C recipes
b2a9826 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Stage C confirmations (W32/64/96 tuned lr) + W128 at lr 2e-3
d69acba 
Full-scale runs on the unchanged 100-epoch cosine so val curves compare
against incumbents at matched epochs. W128 logs to its own project
(mp-gga-ggau-w128-lr2e3) to stay separate from the live 1e-3 run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Reconcile resume checkpoints against the bucket with --update
a04d07c 
A preemption retry can land on a node whose local last.ckpt is days
stale; trusting local copies unconditionally rolled W160 back six
epochs. Always fetch bucket last*.ckpt with --update so the newer of
local/bucket wins before promotion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Fix ckpt_path in stage-C/W128 tuned configs: match run_training.sh st…
9f0a387 
…em strip

run_training.sh derives CKPT_DIR/CKPT_S3 from the config stem AFTER
stripping the config_ prefix; these four set ckpt_path with the prefix,
so Lightning wrote checkpoints the wrapper never synced or restored —
preemption retries cold-started and all weights stayed node-local.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
Refuse resume when local last.ckpt is staler than the bucket's
946c43d 
A transient bucket-fetch failure (swallowed by || true) let a stale node
checkpoint win promotion and silently rolled W128 back ~20 epochs. Guard
compares bucket vs local mtimes post-promotion and fails the attempt so
the retry refetches.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@forklady42 @claude
W128 withheld-test-set eval: 0.481% combined NMAE (GGA 0.480 / GGA+U …
b142576 
…0.483)

Evaluate w128_ckpt_epoch55_val0.005785.ckpt on the test split of
split_capped.json (1700 GGA + 524 GGA+U), staged from s3://oa-electrai and
verified byte-identical to the training-time splits. Della jobs 12057752 +
12111587, fp32 on one A100-80GB.

Two provenance findings recorded in the doc/config comments:
- The campaign's gga+u inputs are PADS, not SAD (identical filelists and
  labels; SAD inputs score ~10% NMAE across the whole subset).
- bf16-mixed autocast OOMs W128 inference on A100 (~75 GiB decoder-conv
  allocation on a 108^3 sample); evaluate at fp32.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@forklady42