Skip to content

About

No description, website, or topics provided.

Resources

Stars

15 stars

Watchers

0 watching

Forks

Repository files navigation

Learnable Infinite Taylor Gaussian for Dynamic View Rendering

teaser

📖 Motivation and Abstract

Instead of relying on time-conditioned polynomial functions to approximate Gaussian trajectories and directions, this solution investigates a more accurate Gaussian evolution model for dynamic scenarios.

Capturing the temporal evolution of Gaussian properties such as position, rotation, and scale is a challenging task due to the vast number of time-varying parameters and the limited photometric data available, which generally results in convergence issues, making it difficult to find an optimal solution. While feeding all inputs into an end-to-end neural network can effectively model complex temporal dynamics, this approach lacks explicit supervision and struggles to generate high-quality transformation fields. On the other hand, using time-conditioned polynomial functions to model Gaussian trajectories and orientations provides a more explicit and interpretable solution, but requires significant handcrafted effort and lacks generalizability across diverse scenes. To overcome these limitations, this paper introduces a novel approach based on a learnable infinite Taylor Formula to model the temporal evolution of Gaussians. This method offers both the flexibility of an implicit network-based approach and the interpretability of explicit polynomial functions, allowing for more robust and generalizable modeling of Gaussian dynamics across various dynamic scenes.

📋 TODO List

  • Repo - Create repo for TaylorGaussian.
  • Clean - Clean the system
  • Test - Test the system on different Ubuntu servershub
  • Code - Release code to the community

🔧 Installation

1. Clone the TaylorGaussian Repo

git clone https://github.com/EllisonKing/taylor_gaussian.git
cd taylor_gaussian
conda create -n taylorgaussian python=3.10
conda activate taylorgaussian

2. Environment Setup

pip install -r requirements.txt
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
pip install setuptools==69.5.1  
pip install numpy==1.23.5
pip install Cmake

pip install git+https://github.com/Po-Hsun-Su/pytorch-ssim.git
pip install git+https://github.com/facebookresearch/pytorch3d.git

Additionally, please install diff-gaussian-rasterization and simple-knn following commands:

cd submodules
git clone https://github.com/ashawkey/diff-gaussian-rasterization.git
cd ./diff-gaussian-rasterization/
pip install -e .
cd ..
git clone https://github.com/camenduru/simple-knn.git
cd ./simple-knn/
pip install -e .

💾 Datasets

1. N3DV

Download the dataset from here.

python script/n3d_process.py --videopath ./data/Neural3D/cook_spinach

For each sequence, the structure is organized as follows:

cook_spinach
├── cam00
├── cam01
├── cam02
├── cam<....>
├── colmap_0
├── colmap_1
├── colmap_2
├── colmap_3
├── colmap_4
├── colmap_<....>
└── poses_bounds.npy

2. Technicolor

The dataset is provided by: Dataset and Pipeline for Multi-View Light-Field Video,

python script/technicolor_process.py --videopath ./data/Technicolor/Birthday

For each sequence, the structure is organized as follows:

Birthday
├── Birthday_undist_00001_00.png
├── Birthday_undist_00001_01.png
├── Birthday_undist_00001_02.png
├── Birthday_undist_<....>
├── cameras_parameters.txt
├── colmap_0
├── colmap_1
├── colmap_<....>

📊 Results

1. SOTA Performance

Table I. Comparison of rendering on the NV3D dataset. ↓ indicates lower is better, ↑ indicates higher is better.
Squence
Cook Spinach
Sear steak
Flame Steak
Cut Roast Beef
Flame Salmon
Coffee Martini
Metric PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓
D3DGS 20.53 0.881 0.153 25.02 0.944 0.072 23.02 0.919 0.113 22.35 0.907 0.125 23.03 0.863 0.153 23.42 0.865 0.138
4DGS 28.12 0.940 0.038 29.07 0.957 0.028 25.04 0.918 0.079 29.71 0.944 0.033 27.90 0.916 0.063 28.18 0.924 0.049
FSGS 29.60 0.919 0.115 29.87 0.945 0.116 29.42 0.943 0.113 28.46 0.913 0.122 28.06 0.926 0.102 26.57 0.907 0.169
SCGS 17.20 0.734 0.232 28.77 0.951 0.056 23.49 0.902 0.104 6.29 0.007 0.683 5.63 0.002 0.681 22.81 0.883 0.142
Ours 32.59 0.966 0.054 33.12 0.973 0.049 33.34 0.971 0.052 33.06 0.969 0.055 28.93 0.951 0.089 27.51 0.945 0.088
Table II. Methods comparison on the Technicolor dataset. ↓ indicates lower is better, ↑ indicates higher is better.
Method
Birthday
Painter
Train
Fatma
PSNR↑
SSIM↑
LPIPS↓
PSNR↑
SSIM↑
LPIPS↓
PSNR↑
SSIM↑
LPIPS↓
PSNR↑
SSIM↑
LPIPS↓
D3DGS 33.810.9650.014 37.380.9570.036 --- 38.400.9110.093
STG 33.870.9510.038 37.300.9280.095 33.360.9480.036 37.280.9060.155
FSGS 26.260.9200.068 34.360.9580.063 30.390.9650.032 27.620.8250.276
4DGS 21.940.9020.071 28.610.9400.058 22.360.8780.124 23.420.7630.236
Ours 34.720.9880.013 38.370.9850.022 35.300.9900.008 38.910.9450.071

2. Erratum

The experimental results on N3DV Dataset shown in Table 1 of our first arXiv paper are erroneous due to our oversight in setting eval=false during training. This oversight resulted in the test images being unintentionally exposed during training. We have since fixed the bug and the correct results are shown in Table 1 of our latest arXiv paper. If you are using the same code for testing, please ensure that the correct setting is used. For detailed experimental logs, please refer here.

⭕️ Acknowledgement

This work incorporates many open-source codes. We extend our gratitude to the authors of the software.

✉️ Citation

If you find this code/work useful for your own research, please consider citing:

@inproceedings{hu2025learnable,
  title={Learnable Infinite Taylor Gaussian for Dynamic View Rendering},
  author={Hu, Bingbing and Li, Yanyan and Xie, Rui and Xu, Bo and Dong, Haoye and Yao, Junfeng and Lee, Gim Hee},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={26844--26854},
  year={2025}
}

About

No description, website, or topics provided.

Resources

Stars

15 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages