Instead of relying on time-conditioned polynomial functions to approximate Gaussian trajectories and directions, this solution investigates a more accurate Gaussian evolution model for dynamic scenarios.
- Repo - Create repo for TaylorGaussian.
- Clean - Clean the system
- Test - Test the system on different Ubuntu servershub
- Code - Release code to the community
git clone https://github.com/EllisonKing/taylor_gaussian.git
cd taylor_gaussian
conda create -n taylorgaussian python=3.10
conda activate taylorgaussian
pip install -r requirements.txt
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
pip install setuptools==69.5.1
pip install numpy==1.23.5
pip install Cmake
pip install git+https://github.com/Po-Hsun-Su/pytorch-ssim.git
pip install git+https://github.com/facebookresearch/pytorch3d.git
Additionally, please install diff-gaussian-rasterization and simple-knn following commands:
cd submodules
git clone https://github.com/ashawkey/diff-gaussian-rasterization.git
cd ./diff-gaussian-rasterization/
pip install -e .
cd ..
git clone https://github.com/camenduru/simple-knn.git
cd ./simple-knn/
pip install -e .
Download the dataset from here.
python script/n3d_process.py --videopath ./data/Neural3D/cook_spinach
For each sequence, the structure is organized as follows:
cook_spinach
├── cam00
├── cam01
├── cam02
├── cam<....>
├── colmap_0
├── colmap_1
├── colmap_2
├── colmap_3
├── colmap_4
├── colmap_<....>
└── poses_bounds.npy
The dataset is provided by: Dataset and Pipeline for Multi-View Light-Field Video,
python script/technicolor_process.py --videopath ./data/Technicolor/Birthday
For each sequence, the structure is organized as follows:
Birthday
├── Birthday_undist_00001_00.png
├── Birthday_undist_00001_01.png
├── Birthday_undist_00001_02.png
├── Birthday_undist_<....>
├── cameras_parameters.txt
├── colmap_0
├── colmap_1
├── colmap_<....>
Squence |
Cook Spinach |
Sear steak |
Flame Steak |
Cut Roast Beef |
Flame Salmon |
Coffee Martini |
||||||||||||
| Metric | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ |
| D3DGS | 20.53 | 0.881 | 0.153 | 25.02 | 0.944 | 0.072 | 23.02 | 0.919 | 0.113 | 22.35 | 0.907 | 0.125 | 23.03 | 0.863 | 0.153 | 23.42 | 0.865 | 0.138 |
| 4DGS | 28.12 | 0.940 | 0.038 | 29.07 | 0.957 | 0.028 | 25.04 | 0.918 | 0.079 | 29.71 | 0.944 | 0.033 | 27.90 | 0.916 | 0.063 | 28.18 | 0.924 | 0.049 |
| FSGS | 29.60 | 0.919 | 0.115 | 29.87 | 0.945 | 0.116 | 29.42 | 0.943 | 0.113 | 28.46 | 0.913 | 0.122 | 28.06 | 0.926 | 0.102 | 26.57 | 0.907 | 0.169 |
| SCGS | 17.20 | 0.734 | 0.232 | 28.77 | 0.951 | 0.056 | 23.49 | 0.902 | 0.104 | 6.29 | 0.007 | 0.683 | 5.63 | 0.002 | 0.681 | 22.81 | 0.883 | 0.142 |
| Ours | 32.59 | 0.966 | 0.054 | 33.12 | 0.973 | 0.049 | 33.34 | 0.971 | 0.052 | 33.06 | 0.969 | 0.055 | 28.93 | 0.951 | 0.089 | 27.51 | 0.945 | 0.088 |
Method |
Birthday |
Painter |
Train |
Fatma |
||||||||
PSNR↑ |
SSIM↑ |
LPIPS↓ |
PSNR↑ |
SSIM↑ |
LPIPS↓ |
PSNR↑ |
SSIM↑ |
LPIPS↓ |
PSNR↑ |
SSIM↑ |
LPIPS↓ |
|
| D3DGS | 33.81 | 0.965 | 0.014 | 37.38 | 0.957 | 0.036 | - | - | - | 38.40 | 0.911 | 0.093 |
| STG | 33.87 | 0.951 | 0.038 | 37.30 | 0.928 | 0.095 | 33.36 | 0.948 | 0.036 | 37.28 | 0.906 | 0.155 |
| FSGS | 26.26 | 0.920 | 0.068 | 34.36 | 0.958 | 0.063 | 30.39 | 0.965 | 0.032 | 27.62 | 0.825 | 0.276 |
| 4DGS | 21.94 | 0.902 | 0.071 | 28.61 | 0.940 | 0.058 | 22.36 | 0.878 | 0.124 | 23.42 | 0.763 | 0.236 |
| Ours | 34.72 | 0.988 | 0.013 | 38.37 | 0.985 | 0.022 | 35.30 | 0.990 | 0.008 | 38.91 | 0.945 | 0.071 |
The experimental results on N3DV Dataset shown in Table 1 of our first arXiv paper are erroneous due to our oversight in setting eval=false during training. This oversight resulted in the test images being unintentionally exposed during training. We have since fixed the bug and the correct results are shown in Table 1 of our latest arXiv paper. If you are using the same code for testing, please ensure that the correct setting is used. For detailed experimental logs, please refer here.
This work incorporates many open-source codes. We extend our gratitude to the authors of the software.
If you find this code/work useful for your own research, please consider citing:
@inproceedings{hu2025learnable,
title={Learnable Infinite Taylor Gaussian for Dynamic View Rendering},
author={Hu, Bingbing and Li, Yanyan and Xie, Rui and Xu, Bo and Dong, Haoye and Yao, Junfeng and Lee, Gim Hee},
booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
pages={26844--26854},
year={2025}
}