End-to-end, one-command pipeline that turns the zavod70 dataset (126 DJI drone stills of an abandoned factory, 4000×3000, ~1 fps oblique flyover) into a 3D Gaussian Splatting scene and renders novel camera paths from it, using NVIDIA ViPE for camera poses, intrinsics and metric depth.
raw stills ──▶ prepare ──▶ ViPE ──▶ COLMAP export ──▶ gsplat training ──▶ rendered fly-through
(Drive) resize/rename poses+depth seed points 3D Gaussians refly + orbit + GT compare
Demo video (3 min, full pipeline end-to-end): https://youtu.be/4A6FeBSeERA Results of the verified end-to-end run (free Colab T4): results/zavod70 — the pipeline's
REPORT.md, metrics, held-out input-vs-render pairs, the rendered fly-throughs, camera paths and logs.
- What this does
- Results (verified run)
- Quickstart (RunPod, local CUDA box, Colab)
- Pipeline stages
- Outputs
- Configuration
- Development, tests, CI
- Design decisions
- Troubleshooting
- Assignment deliverables
- Repository layout
- Licenses and acknowledgements
ViPE estimates camera intrinsics, camera motion and dense near-metric depth from raw frames — it does not produce splats. This repo wraps ViPE in a reproducible six-stage pipeline (each stage is a CLI sub-command, resumable, driven by one YAML file):
| # | Stage | GPU | What happens |
|---|---|---|---|
| 0 | download |
– | Fetch the 126 raw frames from the assignment's Google Drive folder using a checked-in manifest of file ids (parallel, resumable, verified). |
| 1 | prepare |
– | Select / resize / rename frames into the directory layout ViPE consumes (frame_000000.jpg, …). 12 MP originals are reduced to 1920×1440 (1024×768 on a 12 GB-RAM machine): ViPE caches every frame of the sequence in host RAM. |
| 2 | vipe |
✔ | Run ViPE (default pipeline: GeoCalib intrinsics → DROID-style SLAM with bundle adjustment → UniDepth + Video-Depth-Anything metric depth). Also exports the sparse SLAM map and a preview video. |
| 3 | convert |
– | Export ViPE's artifacts to a COLMAP text model (cameras.txt, images.txt, points3D.txt) plus pristine images. Seed points come from the SLAM map (or dense-depth unprojection as fallback). |
| 4 | train |
✔ | Optimise 3D Gaussians with gsplat (L1 + D-SSIM, SH warm-up, adaptive densification). Every 8th frame is held out → honest PSNR/SSIM. Writes splats.ply (opens in SuperSplat / any 3DGS viewer). |
| 5 | render |
✔ | Render camera paths to MP4: a smoothed re-flight of the drone trajectory, a 360° orbit around the scene, and an input-vs-render side-by-side over the original cameras. |
Everything that does not need CUDA (stages 0, 1, 3, all geometry, the training loop's logic, video encoding) is covered by a CPU test-suite that runs in CI; the CUDA kernels themselves come from NVIDIA (ViPE) and nerfstudio (gsplat).
vipe-gs run-all was executed end-to-end on the real dataset on a free Google Colab T4
(15 GB VRAM, 12.7 GB RAM — hence 1024×768 frames, see docs/COLAB.md). Everything in
results/zavod70 was produced by that run; nothing was hand-edited.
| Stage | Time on T4 | Output |
|---|---|---|
| download | 1m05s | 126 JPEGs verified at 4000×3000 |
| prepare | 1m25s | 126 frames at 1024×768 |
| vipe | 6m03s | poses for 126/126 frames, fx = fy = 765.6 px (HFOV 67.5°), metric depth, SLAM map |
| convert | 5.6s | COLMAP model, 312 923 seed points |
| train | 21m06s | 3.39 M Gaussians, 10 000 steps |
| render | 3m30s | refly + orbit (2 × 240 frames) + GT-vs-render over all 126 cameras |
| total | 33m14s | (+ ≈ 28 min one-off environment build) |
Quality on the 16 held-out views (every 8th frame, never seen in training): PSNR 15.3 dB,
SSIM 0.39; training views reach 21–23 dB. The spread is large — 25.3 dB on the frame over the
brick building, 13–17 dB on frames dominated by leafless trees, where 10 k steps of a 3.4 M-splat
model leave the vegetation blurry/wispy. Structures (roofs, walls, roads, tyre stacks, shadows) are
reconstructed sharply — see the pairs in results/zavod70/eval/ and the videos in
results/zavod70/videos/. Straightforward ways to raise the numbers, all
supported by the config: 30 k steps and 1920 px frames on a 24 GB GPU
(-s train.max_steps=30000, default prepare.max_long_side), and — as code extensions listed in
docs/DESIGN.md — camera-pose refinement and per-image exposure embeddings.
-
Create a pod from the
runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04template (the devel image shipsnvcc, which ViPE and gsplat need to compile their CUDA extensions). Give it ≥ 60 GB of volume disk and open the web terminal / SSH. Detailed walkthrough: docs/RUNPOD.md. -
Clone and bootstrap (≈ 15 min, mostly compiling ViPE):
cd /workspace git clone https://github.com/mammadalimammadaliyevpython-dot/vipe-gs.git && cd vipe-gs bash scripts/setup_env.sh # apt ffmpeg, pip install -e ., nvidia-vipe 1.2.0, gsplat 1.5.3, kernel warm-up vipe-gs doctor # python / torch / CUDA / GPU / vipe / gsplat / ffmpeg checks
-
Run the whole pipeline (≈ 30–45 min on a 4090, resumable — re-running skips finished stages):
vipe-gs run-all # uses configs/zavod70.yaml -
Look at the results:
cat runs/zavod70/REPORT.md # timings, intrinsics, #Gaussians, PSNR/SSIM ls runs/zavod70/05_render/ # refly.mp4 orbit.mp4 demo_trajectory.mp4 comparison_gt_vs_render.mp4 python scripts/build_demo_video.py runs/zavod70 # captioned demo video (log replay, 3D view, renders) -> demo.mp4
Download
runs/zavod70/04_gsplat/splats.plyand drop it into https://superspl.at/editor to inspect the splats interactively.
Same as above; you need a CUDA build of PyTorch ≥ 2.4 and a matching nvcc on PATH
(pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128 for CUDA 12.8).
scripts/setup_env.sh checks both before it starts compiling. Or build the Docker image:
docker build -t vipe-gs .
docker run --gpus all --rm -it -v "$PWD/runs:/workspace/vipe-gs/runs" vipe-gs vipe-gs run-alldocs/COLAB.md has the cells. The only differences to A/B: -s prepare.max_long_side=1024
(12.7 GB host RAM) and -s vipe.save_viz=false (2 vCPUs); total ≈ 1 h including the environment build.
vipe-gs download && vipe-gs prepare # CPU, ~1 min
vipe-gs vipe # GPU, ~10-15 min (first run also downloads ~4 GB of model weights)
vipe-gs convert # CPU, seconds
vipe-gs train # GPU, ~7 min for 10k steps at 1920x1440 on a 4090
vipe-gs render # GPU, ~2 min
vipe-gs report # regenerate REPORT.mdAny stage accepts --force to redo it; run-all --from convert restarts from a given stage.
Configuration values can be overridden inline, e.g. vipe-gs -s train.max_steps=30000 -s render.seconds_per_path=12 run-all.
Google Drive folder listings are unreliable for scripted use (rate limits, 50-item view caps), so
data/zavod70_manifest.json pins every file name and Drive id. The
stage downloads in parallel with retries, verifies each JPEG decodes at 4000×3000, and skips files
already present. If you have the frames locally, set dataset.local_dir and this stage is a no-op.
ViPE accepts an MP4 or a directory of equally-sized frames. We use the directory route (no
re-encoding loss) and produce runs/zavod70/01_frames/frame_%06d.jpg plus a manifest mapping each
output back to its source. The long side is resized to 1920 px (config prepare.max_long_side)
because ViPE caches the whole sequence — three times over — as float32 RGB in host RAM and moves
frames to the GPU one at a time: 126 × 4000×3000 would need > 50 GB, 1920×1440 ≈ 14 GB (fine on a
RunPod pod, too much for a 12 GB Colab VM, where 1024 px is the working setting — measured numbers in
docs/DESIGN.md). EXIF orientation is baked in,
dimensions are kept even for H.264, and frame_stride/frame_start/frame_end allow sub-sampling.
Frame index == ViPE stream index, which stage 3 relies on.
Equivalent to vipe infer --image-dir runs/zavod70/01_frames, invoked through ViPE's Python API so
we can add the Hydra overrides the CLI does not expose:
pipeline=default pipeline.output.save_artifacts=true pipeline.output.save_slam_map=true
pipeline.output.save_viz=true pipeline.init.instance=null streams=frame_dir_stream streams.base_path=…
save_slam_map=truekeeps the sparse, multi-view-consistent SLAM point cloud — the best seed for Gaussian Splatting (this is what ViPE's ownvipe_to_colmap.py --use_slam_mapuses).init.instance=nullskips GroundingDINO + SAM instance/sky masking: the factory scene is static and the oblique views contain no sky. Flipvipe.mask_dynamic_objects: truefor footage with people/vehicles/sky.- Output:
pose/*.npz(camera-to-world, OpenCV axes),intrinsics/*.npz(fx fy cx cy),depth/*.zip(EXR, metres),rgb/*.mp4,vipe/*_slam_map.pt,vipe/*_vis.mp4(rgb | depth | point-cloud preview).
A self-contained converter (no vipe import needed) writes the standard COLMAP text model that every
3DGS implementation reads. Poses are inverted to world-to-camera and stored as qw qx qy qz tx ty tz;
one PINHOLE camera is written (ViPE optimises a single intrinsic set per sequence — the converter
checks that and uses the median). Images are copied from the prepared frames (exact ViPE inputs)
instead of being re-decoded from ViPE's H.264 preview. Seed points: SLAM map when present, otherwise
dense depth unprojection with ViPE's local-consistency reliability mask, capped at convert.max_points.
export_summary.json records the horizontal FOV — a quick sanity check for GeoCalib's intrinsics
(DJI wide cameras are ≈ 70–84°).
A compact trainer (src/vipe_gs/train.py, ~200 lines) on top of gsplat's rasteriser and
DefaultStrategy, following the original 3DGS recipe: per-parameter Adam, position LR scaled by the
scene extent and decayed exponentially to 1 %, SH degree ramp every 1000 steps, loss
0.8·L1 + 0.2·(1−SSIM), densify/prune every 100 steps during the first half of training (500 → 5 000
of 10 000 steps, the reference's 15k-of-30k ratio), opacity reset every 3000. Densification is
scene-driven — this textured aerial scene grows from 0.3 M seeds to ≈ 3.5 M Gaussians — so
train.max_gaussians can cap it to a VRAM budget (≈ 1.7 kB per Gaussian while training). Images stay
on the GPU as uint8 (≈ 1 GB) and are converted per step. Every 8th view is held
out; PSNR/SSIM on that split, a CSV training log, side-by-side eval PNGs, a checkpoint and a
viewer-compatible splats.ply are written. The rasteriser is injected, so the loop — including the
real gsplat densification code — is unit-tested on CPU with a fake rasteriser.
- refly – the drone's own trajectory, Gaussian-smoothed and re-sampled at uniform speed (cubic spline for positions, SLERP for orientations). Stays inside the observed volume, so it is the fairest showcase of reconstruction quality.
- orbit – a circle around the scene centre at the flight's median height and look-down angle, revealing structure from directions the drone never flew. The scene's "up" is the normal of the dominant plane of the point cloud (PCA) — robust for aerial captures.
- comparison –
[input frame | 3DGS render]on the original cameras.
Camera paths are saved as JSON next to the videos so any render is reproducible or hand-editable.
runs/zavod70/
├── 00_raw/ 126 original JPEGs (skipped when dataset.local_dir is set)
├── 01_frames/ frame_000000.jpg … + frames_manifest.json
├── 02_vipe/ ViPE artifacts (pose/ intrinsics/ depth/ rgb/ vipe/) + vipe_summary.json
├── 03_colmap/ images/ sparse/0/{cameras,images,points3D}.txt export_summary.json
├── 04_gsplat/ splats.ply checkpoint_final.pt metrics.json train_log.csv eval/*.png
├── 05_render/ refly.mp4 orbit.mp4 demo_trajectory.mp4 comparison_gt_vs_render.mp4 camera_path_*.json
├── REPORT.md / pipeline_report.json / config_used.yaml
└── logs/vipe_gs.log
Each stage directory also contains a .done.json marker (duration, outputs) that makes the pipeline resumable.
One file drives everything: configs/zavod70.yaml. Every field is
documented in src/vipe_gs/config.py (pydantic models reject typos).
Override from the command line with dotted keys:
vipe-gs -s prepare.max_long_side=1600 -s vipe.pipeline=no_vda run-all # smaller GPU
vipe-gs -s train.max_steps=30000 -s train.save_steps=[7000,15000] train --force
vipe-gs -s render.paths=[orbit] -s render.orbit.pitch_deg=55 -s render.seconds_per_path=12 render --force
vipe-gs -c configs/my_scene.yaml -s dataset.local_dir=/data/my_frames run-allpython -m pip install -e ".[dev]" # CPU-only dev install (torch CPU wheel is fine)
make lint # ruff check + format
make test # 70+ tests, ~3 s
BUILD_NO_CUDA=1 pip install gsplat==1.5.3 # optional: lets tests use the real gsplat DefaultStrategy on CPUThe suite builds a synthetic aerial scene with analytic depth (a textured ground plane seen from an arc of oblique cameras), writes it in ViPE's artifact format, and verifies: frame preparation, EXR depth round-trips, COLMAP export (seed points land on the true plane → pose/intrinsics conventions are right), camera-path geometry, PLY/checkpoint round-trips, SSIM/PSNR, MP4 encoding, the CLI's resumability, Hydra-override syntax, and the full train→render flow with a fake rasteriser (including gsplat's real densification/pruning/opacity-reset code). GitHub Actions runs lint + tests on Python 3.10/3.11.
- ViPE is used for what it is — a pose/intrinsics/depth engine — and its native
frame_dir_streaminput avoids a lossy video re-encode. The one preprocessing that is required (downscaling for GPU memory) is scripted and documented with numbers. - COLMAP as the interchange format makes the reconstruction usable with any 3DGS tool
(original 3DGS, nerfstudio
splatfacto, gsplat examples) — not just this trainer. - Own compact trainer instead of
gsplat/examples/simple_trainer.py: the example pulls in ~15 extra dependencies (nerfview, viser, torchmetrics, fused-ssim, ppisp, nvidia-ncore …) that are fragile to install; the library API (rasterization,DefaultStrategy) is stable and small. The trade-off is that we own ~200 lines of optimisation code — mitigated by tests against gsplat's real strategy and by keeping the reference hyper-parameters. - Stages run as separate processes in
run-all, so ViPE's models release GPU memory before training starts, and any stage can be re-run in isolation. - Config as code: one typed YAML, dotted overrides, and a
config_used.yamlnext to the results for reproducibility.REPORT.mdis generated so numbers in the write-up cannot drift from the run. - Honest metrics: a held-out view split is on by default; disable with
train.test_every=0when you want every frame used for the final asset.
| Symptom | Fix |
|---|---|
nvcc not found during setup |
Use a *-devel CUDA image (RunPod PyTorch templates ending in -devel) or install the CUDA toolkit matching torch.version.cuda. |
| ViPE build fails with a torch/CUDA mismatch | python -c "import torch; print(torch.__version__, torch.version.cuda)" and nvcc --version must agree on the CUDA major.minor. Never let pip replace torch: the script uses --no-build-isolation. |
| CUDA OOM in stage 2 | Lower prepare.max_long_side (1600 or 1280) and re-run prepare --force + vipe --force; or use vipe.pipeline: no_vda. |
| Stage 2 dies with exit code −9 (no Python traceback) | The Linux OOM killer: ViPE's frame caches live in host RAM (≈ 3 GB + 36 bytes × frames × pixels). Check `dmesg |
First vipe-gs train sits for ~10 min at 100 % CPU before step 0 |
gsplat is JIT-compiling its CUDA kernels because the cache key (nvcc flags) differs from the warm-up in setup_env.sh. One-time cost; the fixed script compiles with the runtime flags. |
| SLAM fails to initialise / very few poses | Increase overlap: for sparse stills try vipe.extra_overrides: ["pipeline.slam.filter_thresh=1.5", "pipeline.slam.keyframe_thresh=3.0"]. |
Intrinsics look wrong (HFOV in export_summary.json far from 70–84°) |
GeoCalib misfire on textureless frames; re-run ViPE with vipe.pipeline: wide_angle only if the lens really is wide/fisheye, otherwise check frame orientation. |
| CUDA OOM in stage 4 | Cap the model: train.max_gaussians: 3000000 (≈ 5 GB); or train.downscale: 2, train.packed: true. The step log prints the Gaussian count and peak memory every 100 steps. |
| Google Drive quota errors while downloading | Retry later, or download the folder manually / with gdown --folder <url> and set dataset.local_dir. |
| Videos will not play | They are H.264 yuv420p; make sure ffmpeg is on PATH (vipe-gs doctor) — imageio's bundled binary is used otherwise. |
vipe-gs doctor prints the environment state; runs/<name>/logs/vipe_gs.log has every stage's log.
| Deliverable | Where |
|---|---|
| Working ViPE setup (local or RunPod) | scripts/setup_env.sh, Dockerfile, docs/RUNPOD.md, vipe-gs doctor |
| Dataset prepared in ViPE's format (scripted preprocessing) | stages download + prepare, data/zavod70_manifest.json |
| Gaussian Splatting result from the dataset | runs/zavod70/04_gsplat/splats.ply + metrics.json (stage train); measured run: results/zavod70 |
| Rendered trajectory / camera path | runs/zavod70/05_render/*.mp4 + camera_path_*.json (stage render); this run's videos: results/zavod70/videos |
| Public GitHub repo with instructions and scripts | this repository |
| Demo video of the full pipeline | scripts/build_demo_video.py (generated from the run's artifacts) + docs/DEMO_VIDEO.md |
configs/zavod70.yaml pipeline configuration for the assignment dataset
data/zavod70_manifest.json Google Drive ids of the 126 frames
docs/ RUNPOD.md, COLAB.md (step-by-step), DEMO_VIDEO.md (recording guide), DESIGN.md (notes)
results/zavod70/ outputs of the verified Colab T4 run: report, metrics, eval pairs, videos, camera paths, logs
scripts/setup_env.sh CUDA environment bootstrap (RunPod / local)
scripts/build_demo_video.py generate the captioned demo video from a run (log replay, 3D view, renders)
scripts/make_demo_video.sh plain concatenation of the output videos
scripts/collect_results.py curate a run (or its .tar.gz) into a small results/ folder
src/vipe_gs/
cli.py `vipe-gs` command line (stages, run-all, report, doctor)
config.py typed configuration + dotted overrides
download.py prepare.py stages 0-1
vipe_runner.py stage 2 (ViPE Python API + Hydra overrides)
vipe_artifacts.py readers for ViPE's output files
colmap.py stage 3: COLMAP text model I/O + export
scene.py gaussians.py losses.py train.py stage 4
geometry.py trajectory.py video.py render.py stage 5
report.py utils.py
tests/ CPU test-suite (synthetic scene with analytic depth, fake rasteriser)
Dockerfile, Makefile, .github/workflows/ci.yml
This repository's code is MIT licensed (see LICENSE). It orchestrates third-party software with
their own terms: ViPE (Apache-2.0, downloads model weights with
their own licenses — e.g. UniDepth/Video-Depth-Anything; the optional UniK3D component is CC BY-NC-SA)
and gsplat (Apache-2.0). The dataset belongs to the
assignment's authors and is not redistributed here — only its Drive file ids.