Marco Simoni

Ph.D. in Artificial Intelligence — Sapienza University of Rome

Reinforcement Learning for LLM post-training · Foundation Models · AI for Cybersecurity

Research · Projects · Publications · Writing · Contact

I work on making language models reason reliably — from the optimization algorithms that train them to the architectures that run them.

My Ph.D. research (Sapienza University of Rome, in collaboration with CNR-IIT) focused on reinforcement learning for LLM post-training and on adapting foundation models to high-stakes domains, primarily cybersecurity. I design policy optimization methods that stay stable without a reference model, build LLM architectures from scratch, and ship retrieval and graph-reasoning systems that outperform frontier models on domain-specific tasks.

Reproducible code, honest baselines, benchmarks that measure what they claim to measure.

flowchart LR
    A["Pretrained LLM"] --> B["RL Post-Training<br/>GTPO · GRPO · PPO"]
    B --> C["Reasoning Model"]
    C --> D["Retrieval &amp; Graph Grounding<br/>MoRSE · TITAN"]
    D --> E["Reliable Domain Expert<br/>Cybersecurity"]
    F["Architecture from Scratch<br/>MoE · RoPE · GQA"] --> A

    style B fill:#1f6feb,color:#fff,stroke:#1f6feb
    style D fill:#238636,color:#fff,stroke:#238636
    style E fill:#8957e5,color:#fff,stroke:#8957e5

A KL-free policy optimization algorithm for LLM post-training. GTPO identifies conflict tokens — tokens appearing at the same position across completions with opposite rewards — and protects them by skipping negative updates while amplifying positive ones. It additionally filters completions whose entropy exceeds a provable threshold, preventing policy collapse without a reference model.

Evaluated on GSM8K, MATH and AIME2024, GTPO consistently outperforms both GRPO and SFT across settings.

@article{simoni2026gtpo,
  title   = {GTPO: Stabilizing Group Relative Policy Optimization
             via Gradient and Entropy Control},
  author  = {Simoni, Marco and Fontana, Aleksandar and Rossolini, Giulio
             and Saracino, Andrea and Mori, Paolo},
  journal = {Transactions of the Association for Computational Linguistics},
  year    = {2026},
  url     = {https://arxiv.org/abs/2508.03772}
}

The first specialized RAG assistant for cybersecurity: a dual-cascaded framework with 7 parallel retrievers, each specialized on a distinct knowledge source (MITRE, CVE, Metasploit, ExploitDB). Knowledge bases refresh in real time — no retraining required.

Result: +15% response accuracy over GPT-4 on general and multi-hop cybersecurity questions.

Bridges natural-language threat queries and executable reasoning over a structured knowledge graph. An LLM path planner predicts relational paths and starting entities; a graph executor traverses them over a modified MITRE knowledge graph to retrieve the answer set.

Released with the TITAN Dataset: 88,209 examples (74,258 train / 13,951 test) pairing questions with executable reasoning paths and step-by-step Chain-of-Thought explanations.

A full LLM architecture implemented from first principles: Sparse Mixture-of-Experts, RoPE, attention gating, GQA. Hardware throughput maximized via sliding-window attention, static KV caching and gradient checkpointing — built to make every architectural trade-off explicit and measurable.

• GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control M. Simoni, A. Fontana, G. Rossolini, A. Saracino, P. Mori — TACL
• On the Hidden Objective Biases of Group-based Reinforcement Learning A. Fontana, M. Simoni, G. Rossolini, P. Mori, A. Saracino — ACL 2026
• Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost S. Nayab, G. Rossolini, M. Simoni, A. Saracino, G. Buttazzo, et al. — Information Sciences  123 citations
• Toward Reliable and Adaptive Large Language Models in the Cybersecurity Domain M. Simoni — Ph.D. Thesis, Sapienza University of Rome

• TITAN: Graph-Executable Reasoning for Cyber Threat Intelligence M. Simoni, A. Fontana, A. Saracino, P. Mori
• Improving LLM Reasoning for Vulnerability Detection via Group Relative Policy Optimization M. Simoni*, A. Fontana*, G. Rossolini, A. Saracino
• MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generation M. Simoni, A. Saracino, V. P., M. Conti — ACM/SIGAPP SAC 2025, 1213–1222  33 citations

_{* Equal contribution.}

CNR-IIT & NetGroup — AI Researcher Engineered an LLM-driven framework automating the translation of natural-language requirements into structured XACML / U-XACML access control policies, deployable on-device for IoT and smart-home environments.

Horus Project — AI Researcher Architected and trained a Transformer-based foundation model from scratch for proactive cyber-attack prediction.

• GTPO vs GRPO: A Smarter Path to Stable Reasoning LLMs — how conflict masks and entropy regularization address GRPO's gradient conflicts and policy collapse.
• REINFORCE vs. Posterior Token Targets: Two Paths to Steering Language Models — the core mechanics of reshaping per-step token probabilities to steer model behavior.