SwissGov-RSD: A Human-annotated, Cross-lingual Benchmark for the Recognition of Token-level Semantic Differences Between Related Documents

A comprehensive benchmark for evaluating token-level semantic difference recognition across multiple languages including iSTS-RSD (Vamvas and Sennrich, 2023) and SwissGov-RSD (Wastl et al., 2025).

---

Installation

We recommend using Python 3.11 for this project.

Install the required dependencies in two steps:

Step 1: Install main requirements

pip install -r requirements.txt

Step 2: Install RSD-specific requirements

pip install -r rsd/requirements.txt

---

Model Training

Encoder Models

Before training encoder models:

1. Adjust the data path in encoders/finetuning/data
2. Update the output_dir name in encoders/finetuning/run_train_modernbert-large.sh
3. Update the model path in scripts/run_predict_encoders.sh

Then run the training script:

bash encoders/finetuning/run_train_modernbert-large.sh

Large Language Models (LLMs)

Llama Models:

bash llama/run_train_llama_8b.sh

GPT-4o-mini:

Training for GPT-4o-mini was performed using the browser application.

---

Running Predictions

Encoder-based Models

bash scripts/run_predict_encoder.sh

Llama Models

bash scripts/run_predict_llama.sh

OpenAI Models

python scripts/predict_openai.py --model [openai_modelname]

Llama 405B (Fireworks)

python scripts/predict_llama_405b_fireworks

DeepSeek R1

python scripts/predict_deepseekr1.py

---

Evaluation

SwissGov-RSD Evaluation

Evaluate predictions on the SwissGov-RSD benchmark:

python -m scripts.evaluate_predictions_admin [prediction_path_prefix]

Example prediction path prefix:

data/evaluation/encoder_predictions/dev/DiffAlign(model=Alibaba-NLP_gte-multilingual-base, layer=-1_admin_

Use --split to evaluate against a specific gold label split regardless of which predictions file is provided (default: full):

python -m scripts.evaluate_predictions_admin [prediction_path_prefix] --split dev
python -m scripts.evaluate_predictions_admin [prediction_path_prefix] --split test

Special Note for DiffAlign with XLM-R + SimCSE:

When evaluating DiffAlign with a model that can take only relatively short input sequences, e.g. XLM-R + SimCSE for SwissGov, add the --short flag:

python -m scripts.evaluate_predictions_admin [prediction_path_prefix] --short

iSTS-RSD Evaluation

Evaluate predictions on the iSTS-RSD benchmark:

python -m scripts.evaluate_predictions_ists [path_to_preds]

---

Utilities and Tools

Dataset Statistics

Get statistics about the test dataset:

python -m scripts.get_test_data_stats

Label Visualization

Visualize Gold Labels:

Display annotations for a specific sample from the gold standard:

python -m scripts.visualize_labeled_text [path_to_gold] [sample_id] --gold

Example:

• sample_id: admin_de_1
• path_to_gold: Full path to the file containing gold annotations

Visualize Predicted Labels:

Display annotations from model predictions:

python -m scripts.visualize_labeled_text [path_to_preds] [sample_id]

Visualize iSTS-RSD Sample:

Create a visualization example from the iSTS-RSD dataset:

python -m scripts.create_rsd_example

Length Analysis Statistics

Get correlation statistics per bucket for length analysis:

python -m scripts.get_corr_per_bucket [path_to_preds] --get-corr-per-bucket

Example path:

data/evaluation/encoder_predictions/dev/DiffAlign(model=Alibaba-NLP_gte-multilingual-base, layer=-1_admin_

Label Projection

The following scripts are available for label projection and evaluation:

• scripts/project_labels.py - Project labels across languages
• scripts/merge_projected_data.py - Merge projected label data
• scripts/sample_projected_data.py - Sample from projected data

Latency Testing

Test model inference latency:

python -m scripts.latency_test

---

Updates

• 07.07.2026: The SwissGov-RSD paper received an SAC Highlight at ACL 2026! 🏆

• 06.04.2026: The SwissGov-RSD paper was accepted to ACL Main! 🎉

• 19.03.2026: Multi-parallel datasets can be very valuable for translation-related tasks! Find a translation-specific subset of SwissGov here: https://github.com/miwytt/multi-parallel-swissgov/

• 04.03.2026: Due to previously low annotator agreements for the German subset, the German subset has been re-annotated. The previous annotations are preserved at: https://github.com/ZurichNLP/SwissGov-RSD/tree/old-german-annotations. Note that the data used in the experiments and described in the paper is the new re-annotated version.

Citation

If you use this library in your research, please cite the accompanying paper:

@misc{wastl2025swissgovrsdhumanannotatedcrosslingualbenchmark,
      title={SwissGov-RSD: A Human-annotated, Cross-lingual Benchmark for Token-level Recognition of Semantic Differences Between Related Documents}, 
      author={Michelle Wastl and Jannis Vamvas and Rico Sennrich},
      year={2025},
      eprint={2512.07538},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.07538}, 
}