ON-POLICY SELF-DISTILLED SAFETY ALIGNMENT
EOPSA red panda mascot filtering tokens and trimming a rollout

Efficient On-Policy Self-Distilled
Safety Alignment

Qirui Liu1,*Yichen Sun1,*Yan Wang2Yu Mi1Wei Cao1Yue Shen2Zhixuan Chu1,†Kui Ren1

1 The State Key Laboratory of Blockchain and Data Security, Zhejiang University

2 Ant Group

* Equal contribution    † Corresponding author

▤ Paper ↗ Code Models❞ BibTeX

Learn from reliable supervision.
Update safety-critical tokens.

Two coordinated mechanisms make safety alignment more efficient while largely preserving reasoning.
~50%

less rollout computation

~2%

of tokens backpropagated

32B

largest evaluated model

Results across Qwen3 and DeepSeek-R1-Distill models · See the experiments below

AT A GLANCE

Two decisions. One efficient alignment framework.

How far to roll out, and which tokens to distill.

Figure 3. EOPSA framework. Adaptive Rollout Scheduling bounds generation to reliable supervision regimes. Selective Distillation restricts updates to Pivot, Intent, and Risk positions.
THE PAPER

Abstract

On-Policy Self-Distillation (OPSD) has emerged as a promising paradigm for safety alignment, delivering dense, token-level supervision by distilling from a teacher conditioned on refusal-oriented privileged prompts. However, we reveal that this paradigm suffers from critical inefficiencies that degrade both training efficiency and general reasoning capabilities. Specifically, we diagnose two fundamental bottlenecks: (1) supervisory collapse over extended rollouts, where the teacher’s corrective efficacy degrades precipitously as the student’s generation prefix lengthens, injecting noisy gradients into late-stage tokens; and (2) gradient dilution from stylistic shifts, where the distillation objective is dominated by safety-irrelevant stylistic discrepancies induced by privileged prompting, washing out genuine safety signals and impairing base reasoning.

To resolve these issues, we propose Efficient On-Policy Self-Distilled Safety Alignment (EOPSA), which concentrates computational and gradient budgets exclusively on reliably supervised, safety-critical tokens. EOPSA incorporates two coordinated mechanisms: Adaptive Rollout Scheduling, which dynamically bounds the generation horizon guided by Teacher Rescue Rate (TRR), and Selective Distillation, which filters out safety-neutral tokens to restrict gradient updates to safety-pivotal transitions. Extensive evaluations across reasoning models up to 32B parameters demonstrate that EOPSA slashes rollout computation by ~50% and backpropagates through merely ~2% of tokens, consistently outperforming full-token distillation baselines in both safety compliance and reasoning retention.

WHY EOPSA?

Dense supervision has two bottlenecks.

Evidence from the paper’s analysis of teacher reliability and token-level divergence.

01 / SUPERVISORY RELIABILITY

Longer prefixes become harder to rescue.

TRR measures teacher continuation on student prefixes. IDR reverses the roles, measuring student continuation on safe teacher prefixes.

Figure 1. Teacher rescue degrades as unaligned student prefixes lengthen, while longer safe teacher prefixes improve defense inheritance.
02 / SEMANTIC RELEVANCE

High divergence can reflect style.

The high-KL subset contains many function words, identical top-1 predictions, and safety-neutral lexical shifts.

Figure 2(a). Approximately 8% of tokens account for 80% of the distillation signal.
Figure 2(b). Safety-neutral categories make up 71.0% of high-KL positions and 51.7% of their KL mass.
METHOD

Adaptive horizons. Selective gradients.

EOPSA coordinates sequence-level truncation and token-level semantic filtering.

TRAINING LOOP · CONCEPTUAL ANIMATION

Watch one complete EOPSA training loop.

The same student-generated prefixes feed a frozen teacher. EOPSA adds a reliable horizon and a safety-critical token mask.

BASE OPSDFull rolloutAll tokens → reverse KL → student update
EOPSA TRAINING CYCLEVALIDATION → TRAINING → VALIDATION
INPUTPotentially harmful query
TRAINABLEStudent policySamples an on-policy response
Student rolloutbounded by L
ARS: validation TRR ≥ τ sets L
FROZENTeacher policySame student prefix + refusal guidance
Compare top-1 token pairsAt each retained student prefix
SELECTIVE DISTILLATION
The → ButPivotthe → aFunctionmake → checkIntentwant → wantConsistentapproach → handleOtheruser → safetyRisk
Keep safety-criticalFilter neutral shifts
PivotIntentRisk
Masked forward KL1/NK ∑ mt KL(PT ∥ PS)Only selected positions contribute
Backpropagate to the studentTeacher stays frozen · recheck TRR at validation checkpoints
Validation chooses L

TRR is measured at validation checkpoints; the longest reliable candidate caps generation.

01 / 05

The token mixing is a visual aid. Classification uses each student–teacher top-1 pair; no random shuffle is performed in training. Token counts and cutoff shown here are illustrative. Method and Algorithm 1 ↗

01

Adaptive Rollout Scheduling

Select the longest candidate horizon whose validation Teacher Rescue Rate meets the reliability threshold. The default candidates are {128, 256, 1024}, with τ = 0.75.

L = max { l ∈ S : TRRl ≥ τ }
Can the teacher still rescue the prefix?
Frozen teacher + refusal guidance
Short student prefix✓ Safe continuation
T →
Long student prefix× Unsafe continuation
T →
Late supervision can become unreliable
ARS keeps the longest reliable horizon.

Illustrative outcomes, not a rule for every sample. TRR measures rescue reliability over validation examples.

02

Selective Distillation

Use the top-1 student–teacher token pair to classify each position. Retain Pivot, Intent, and Risk; filter Function, Consistent, and Other. Match teacher and student distributions only at selected positions.

PivotIntentRisk
Filter tokens before distillation.
the → aThen → Butwant → wantmake → checkon → oruser → safetyapproach → handlesupports → reinforces
Pivot · Intent · Risk× Neutral masked
≈2%of generated tokens receive distillation updates

Reported training fraction. The example token count above is illustrative.

Figure 4. Training dynamics. Category-wise token volume, token and KL filtering efficiency, and TRR-guided rollout scheduling across training.
✦ Explore the mechanisms interactivelyARS simulator + token inspector ＋
SEQUENCE LEVEL01 — ADAPTIVE ROLLOUT SCHEDULING

Go further.
When it’s reliable.

Teacher Rescue Rate (TRR) asks: can the teacher still steer a student’s unfinished answer toward safety?

At validation checkpoints, ARS selects the longest candidate horizon with TRR ≥ 0.75. As alignment improves, longer horizons can become available.

L = max { l ∈ S : TRRl ≥ τ }
The mechanism, in detail ↗
TEACHER RESCUE RATEL = 128 tokens
1.000.750.500.0001282561024τ = 0.75
Auto evolution
EarlyDevelopingAligned

Conceptual animation · Candidate horizons: 128, 256, 1024. Drag to explore.

TOKEN LEVEL02 — SELECTIVE DISTILLATION

Small changes.
Decisive differences.

A large KL divergence can mean a change in style. EOPSA compares the student’s and teacher’s top-1 predictions and keeps only safety-critical transitions.

Automatic token tour
PivotIntentRiskFiltered out
Only selected positions contribute to the distillation loss.

§4.2 · Appendix G. Token pairs share the same student prefix; selected positions receive distribution-level supervision.

EXPERIMENTS

Safety, reasoning, and efficiency.

Original results from the paper, including over-refusal and uncertainty.

BENCHMARK COMPARISON

Stronger safety. Preserved reasoning.

Explore the reported results, benchmark by benchmark.

Qwen3-4B · Safety score (%) ↑0–100 scale

Swipe to explore all benchmarks →

Hover or focus a bar for the exact reported score.
Table 1 · Benchmark comparison. Scores (%) ↑. Hover for exact values; click the legend to compare methods. Download data ↓
Table 1. Main results from the paper. Qwen3-1.7B, Qwen3-4B, and DeepSeek-R1-Distill-Qwen-7B. Purple marks the best result and blue the second-best, excluding the base model. ↑ / ↓ indicate the preferred direction; Δ is relative to OPSA. Click to enlarge.
QWEN3–4B · AVERAGE SAFETY98.08%

71.37% base · 94.18% OPSD

QWEN3–4B · AVERAGE REASONING74.29%

74.65% base · 73.92% OPSD

QWEN3–4B · OPTIMIZED TOKENS / SAMPLE6.04

643.53 for full-token OPSD

Table 1 · Reasoning is averaged across MATH-500, GPQA-D, HumanEval, and LiveCodeBench.

Compare models and reported averages1.7B → 32B ＋
TABLE 1
EOPSA AVERAGE SAFETY
91.15%

+36.74 points over the base model

AVERAGE REASONING61.73 / 62.79 base

7.41 optimized tokens per sample

Over-refusal: 6.20% EOPSA · 0.20% base (lower is better)

Table 1 · Safety averages four benchmarks. Reasoning averages MATH-500, GPQA-D, HumanEval and LiveCodeBench. Scores are percentages; bars use a 0–100 scale.

COMPUTATIONAL EFFICIENCY

Concentrate the training budget.

ARS reduces autoregressive rollout work. Selective Distillation concentrates gradient updates on safety-critical positions.

−49%rollout time vs. OPSD
−28%training time vs. OPSD

Qwen3-1.7B · Figure 5
Rollout time, training time, and effective data size.

Figure 5. Computational efficiency on Qwen3-1.7B. Compared with OPSD, rollout time decreases from 0.51 h to 0.26 h and training time from 0.29 h to 0.21 h.
QUALITATIVE EXAMPLES

See safety alignment in action.

Two cases from Appendix I: hypothetical framing at 1.7B and a chemistry-related jailbreak at 32B.

RESPONSE STUDIO RESPONSE REPLAY
THE REQUEST

QQwen3-1.7B
Unsafe
EEOPSA · 1.7B
Safe

Response replay · Appendix I ↗

REFERENCE

Cite EOPSA

If you find this work useful for your research, please cite our paper.

BibTeX
@misc{liu2026eopsaefficientonpolicyselfdistilled,
  title={EOPSA: Efficient On-Policy Self-Distilled Safety Alignment},
  author={Qirui Liu and Yichen Sun and Yan Wang and Yu Mi and
          Wei Cao and Yue Shen and Zhixuan Chu and Kui Ren},
  year={2026},
  eprint={2609.34519},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2609.34519}
}
Download paper ↓Official repository ↗

Download figure ↓