Skip to content

README.md

Latest commit

 

History

History
137 lines (110 loc) · 6.18 KB

README.md

File metadata and controls

137 lines (110 loc) · 6.18 KB
Raw

SHARP Image2Splat service

FastAPI wrapper around the single-frame SHARP depth-injection path: RGB image + metric depth EXR → 3D Gaussian Splat .ply. Runs on a CUDA GPU host (built/tested on an RTX 4090). Any HTTP client can call it (e.g. via a SHARP_BACKEND_URL your app reads).

Setup (GPU host)

powershell -ExecutionPolicy Bypass -File service\install.ps1   # idempotent, ~few min first run
powershell -ExecutionPolicy Bypass -File service\run.ps1       # starts the server

install.ps1 clones apple/ml-sharp (pinned cdb4ddc6) as a sibling, builds a Python 3.13 venv at <repo>/.venv, installs everything, forces the CUDA torch build (ml-sharp's requirements otherwise pull CPU-only torch), and verifies torch.cuda.is_available() + import sharp, sharp_ext.

Auto-start on boot (headless)

Registered as a Windows Scheduled Task that starts at boot with no login and restarts on crash. It runs as the user account (not SYSTEM), because the venv is built on a per-user Python that SYSTEM may not be able to launch.

Install (self-elevates via UAC if needed):

powershell -ExecutionPolicy Bypass -File service\install-autostart.ps1

This registers task SHARP Image2Splat (AtStartup, RunLevel Limited, restart every 1 min on failure). _autostart.ps1 is the boot entry point — it runs serve.py with stdout/stderr appended to service\logs\service.log.

Logon type (-LogonType, default Auto):

value behaviour
Auto try S4U, fall back to Password
S4U run whether logged on or not, no password stored (the default path)
Password store the account password in the Task Scheduler vault
System run as SYSTEM (no password; can't reach per-user paths or mapped drives)

S4U is preferred because -User/-Password registration fails with 0x8007052e ("user name or password is incorrect") on Microsoft accounts, PIN / Windows Hello sign-in, and blank-password accounts. The trade-off: an S4U task has no network credentials, so outbound HTTP (model download, serving) works but SMB / mapped drives do not — if you need /generate's server-side output_path to write to a share, re-register with -LogonType Password.

First boot downloads the ~2.6 GB checkpoint during startup; uvicorn binds port 8765 only after the model loads, so /health refuses connections for a few minutes on the very first run. Watch progress with:

Get-Content service\logs\service.log -Tail 20 -Wait

Manage it:

Start-ScheduledTask  -TaskName "SHARP Image2Splat"   # start now
Stop-ScheduledTask   -TaskName "SHARP Image2Splat"   # stop
Get-ScheduledTask    -TaskName "SHARP Image2Splat" | Get-ScheduledTaskInfo
Unregister-ScheduledTask -TaskName "SHARP Image2Splat"   # remove auto-start

Redeploy after a code change (e.g. pulling a new revision) — you must Stop then Start:

Stop-ScheduledTask  -TaskName "SHARP Image2Splat"
Start-ScheduledTask -TaskName "SHARP Image2Splat"

The task runs in a non-interactive session, so Stop-Process/Task-Manager from your logged-in session can't see it, and Start-ScheduledTask alone is a no-op while it's already running (MultipleInstances=IgnoreNew). Use Stop-ScheduledTask to terminate it.

Note: the task and run.ps1 both bind port 8765 — don't run both at once.

Config (env)

var default meaning
SHARP_HOST 0.0.0.0 bind address
SHARP_PORT 8765 bind port
SHARP_DEVICE cuda torch device (startup fails loud if cuda requested but unavailable)

Single worker by design — one model instance holds ~6 GB VRAM and is reused across requests.

Endpoints

GET /health

{ "ok": true, "cuda": true, "device": "cuda", "model_loaded": true }

POST /inspect — multipart

Field depth (EXR). Returns depth stats (shape, min/max/mean, sky/far %, finiteness) — use to sanity-check a depth pass before generating.

POST /generate — multipart

field type notes
image file (PNG/JPG) RGB; alpha stripped
depth file (EXR), optional camera-space Z, meters, +forward. Required for exr_pixel/exr_grade; ignored for sharp
albedo file (PNG/JPG), optional Houdini albedo AOV; required only for exr_grade + grade_source=region
f_px float (optional) focal length in px; OR provide the next two
focal_mm + aperture_mm float (optional) f_px = focal_mm/aperture_mm * image_width
depth_method str (default exr_pixel) sharp | exr_pixel | exr_grade (see below)
grade_curve str (default affine) exr_grade only: affine | polynomial | histogram
grade_source str (default percentile) exr_grade only: percentile | region
grade_min_slope float (default 0.0) exr_grade only: floor on the grade-curve slope to stop flicker/popping under camera motion. 0.3–0.6 recommended if you see it
blend_alpha float (default 0.4) exr_pixel only: 0=trust depth, 1=vanilla SHARP
output_path str (optional) if set, write the PLY server-side and return JSON instead of bytes

depth_method:

  • sharp — plain SHARP, no depth injection (no depth needed).
  • exr_pixel — per-pixel inverse-depth blend of the EXR (current behaviour).
  • exr_grade — globally remap SHARP's own predicted depth to match the EXR's distribution (colour-grade style; immune to RGB↔EXR drift). grade_source=percentile matches value percentiles; grade_source=region matches per-region medians using the albedo AOV for segmentation (requires the albedo upload).

Response header X-Depth-Method echoes the method used.

Response: the .ply bytes (application/octet-stream, headers X-Num-Gaussians, X-F-Px), or JSON {ok, path, num_gaussians, f_px, image_size} when output_path is given.

Example:

curl -s http://<gpu-host>:8765/generate \
  -F image=@beauty.png -F depth=@depth.exr \
  -F focal_mm=24 -F aperture_mm=35 -F blend_alpha=0.4 \
  -o splat.ply

Wiring a client

Point your client at the service — e.g. an env var it reads:

SHARP_BACKEND_URL=http://<gpu-host>:8765