Variational Model Merging for Pareto Front Estimation in Multitask Finetuning

This repository contains code for the paper:

Variational Model Merging for Pareto Front Estimation in Multitask Finetuning

Hugo Monzón Maldonado, Nico Daheim, Thomas Möllenhoff, Iryna Gurevych, Mohammad Emtiyaz Khan

https://arxiv.org/abs/2412.08147v2

Mixture-of-Gaussian Merging

The mixture-of-Gaussian merging is currently implemented in merging.py and can be found in the function soupmerging.

Citation

Please use the following citation:

@misc{maldonado2026variationalmodelmergingpareto,
      title={Variational Model Merging for Pareto Front Estimation in Multitask Finetuning}, 
      author={Hugo Monzón Maldonado and Nico Daheim and Thomas Möllenhoff and Iryna Gurevych and Mohammad Emtiyaz Khan},
      year={2026},
      eprint={2412.08147},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2412.08147}, 
}