AESplat: Advancing Pose-free Feed-forward 3D Gaussian Splatting via Decoupled Appearance Modeling

🌐 Project Page | 🤗 Model Weights | 📄 Paper

🚀 What Makes AESplat Special?

AESplat is a pose-free feed-forward 3DGS framework that shifts from a unified appearance modeling paradigm to a decoupled one, substantially improving rendering quality. It stands out with:

• 🔀 Unified → Decoupled: Revisits Gaussian appearance through the lens of spherical harmonics (SH), modeling view-independent and view-dependent appearance with tailored strategies instead of predicting all SH coefficients uniformly.
• 🎨 Efficient Decoupled Appearance Modeling: Zeroth-order SH (view-independent) is derived from input images without training via I2DC; higher-order SH (view-dependent) is predicted by a shallow MLP conditioned on two 3D-aware inductive biases, GVSRE and WCM.
• 📈 SOTA Rendering Quality: Consistently outperforms both pose-free and pose-required feed-forward 3DGS methods—e.g., +0.8 dB PSNR over NAS3R and +1.1 dB over DepthSplat on RealEstate10K—with strong zero-shot generalization to ACID, DL3DV, and ScanNet++.

🚀 Quick Start

📥 Clone the Repository

git clone https://github.com/aesplat/AESplat.git
cd AESplat

⚙️ Environment Setup

conda create -n aesplat python=3.12 -y
conda activate aesplat
pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt --no-build-isolation
pip install submodules/diff-gaussian-rasterization --no-build-isolation

🤗 Checkpoints

Place the weights in checkpoints/.

Checkpoint	Resolution	Trained on	Views	Use it for
AESplat_RE10K.ckpt	224×224	RealEstate10K	2	RealEstate10K, and zero-shot ACID / DL3DV / ScanNet++
AESplat_ACID.ckpt	224×224	ACID	2	In-domain ACID

📂 Dataset Organization

We train on RealEstate10K and ACID, and evaluate zero-shot generalization on ACID, DL3DV, and ScanNet++. Preparation details are in DATASETS.md. The expected layout is:

datasets/
├── re10k/
│   ├── train/
│   └── test/
├── acid/
│   ├── train/
│   └── test/
├── dl3dv/
│   └── test/
└── scannetpp/
    └── test/

Override the default roots when your data lives somewhere else:

dataset.re10k.roots=[YOUR_RE10K_OR_ACID_PATH]
dataset.dl3dv.roots=[YOUR_DL3DV_PATH]
dataset.scannetpp.roots=[YOUR_SCANNETPP_PATH]

🎯 Training & Evaluation

Training

# RealEstate10K
python -m src.main +experiment=aesplat/re10k \
    wandb.mode=online wandb.name=aesplat_re10k

# ACID
python -m src.main +experiment=aesplat/acid \
    wandb.mode=online wandb.name=aesplat_acid \
    dataset.re10k.roots=[YOUR_ACID_PATH]

Evaluation

In-domain evaluation and zero-shot generalization (RE10K → ACID / DL3DV / ScanNet++):

# RealEstate10K (in-domain)
python -m src.main +experiment=aesplat/re10k mode=test wandb.name=re10k \
    dataset/view_sampler@dataset.re10k.view_sampler=evaluation \
    dataset.re10k.view_sampler.index_path=assets/evaluation_index_re10k.json \
    checkpointing.load=./checkpoints/AESplat_RE10K.ckpt \
    test.save_image=false test.save_video=false test.align_pose=false

# ACID (in-domain)
python -m src.main +experiment=aesplat/acid mode=test wandb.name=acid \
    dataset/view_sampler@dataset.re10k.view_sampler=evaluation \
    dataset.re10k.view_sampler.index_path=assets/evaluation_index_acid.json \
    dataset.re10k.roots=[YOUR_ACID_PATH] \
    checkpointing.load=./checkpoints/AESplat_ACID.ckpt \
    test.save_image=false test.save_video=false test.align_pose=false

# RealEstate10K → ACID
python -m src.main +experiment=aesplat/acid mode=test wandb.name=re10k_acid \
    dataset/view_sampler@dataset.re10k.view_sampler=evaluation \
    dataset.re10k.view_sampler.index_path=assets/evaluation_index_acid.json \
    dataset.re10k.roots=[YOUR_ACID_PATH] \
    checkpointing.load=./checkpoints/AESplat_RE10K.ckpt \
    test.save_image=false test.save_video=false test.align_pose=false

# RealEstate10K → DL3DV
python -m src.main +experiment=aesplat/dl3dv mode=test wandb.name=dl3dv \
    dataset/view_sampler@dataset.dl3dv.view_sampler=evaluation \
    dataset.dl3dv.view_sampler.index_path=assets/evaluation_index_dl3dv.json \
    checkpointing.load=./checkpoints/AESplat_RE10K.ckpt \
    test.save_image=false test.save_video=false test.align_pose=false

# RealEstate10K → ScanNet++
python -m src.main +experiment=aesplat/scannetpp mode=test wandb.name=scannetpp \
    dataset/view_sampler@dataset.scannetpp.view_sampler=evaluation \
    dataset.scannetpp.view_sampler.index_path=assets/evaluation_index_scannetpp.json \
    checkpointing.load=./checkpoints/AESplat_RE10K.ckpt \
    test.save_image=false test.save_video=false test.align_pose=false

🎥 Camera Conventions

We follow the pixelSplat camera system.

	Convention
Intrinsics	Normalized. Row 1 is divided by image width, row 2 by image height.
Extrinsics	OpenCV camera-to-world: +X right, +Y down, +Z into the screen.

🙏 Acknowledgements

This project is built upon NAS3R and VGGT. We extend our gratitude to all the authors for their outstanding contributions and excellent repositories!

Citation

If you find this repo useful, please cite:

---

⭐ If AESplat helps your research, please consider starring this repository!

AESplat: Advancing Pose-free Feed-forward 3D Gaussian Splatting via Decoupled Appearance Modeling