DeShortcut-Align

SAFETY ALIGNMENT, BEYOND SURFACE CUES

Decoupling Spurious Shortcuts for Robust Safety Alignment in Large Reasoning Models

Qirui Liu¹Yichen Sun¹Yan Wang²Zhixuan Chu^1,†Linbo JiangJianan Lin³Kui Ren¹

¹ Zhejiang University² Ant Group³ Chongqing Ant Consumer Finance Co., Ltd

† Corresponding author

Read the paper Code Data Explore the method

A safer reasoning model should look past the template and the keyword.
DeShortcut-Align reduces shortcut reliance while preserving the ability to help.

Smaller template gap72.2% ↓

7B · SFT vs. standard SFT

Fewer false rejections58.3% ↓

7B · GRPO vs. standard GRPO

Jailbreak defense97.7%

7B · RL · PAIR / GCG / TAP

Tables 1 & 4 · DeepSeek-R1-Distill-Qwen-7B

01THE MOTIVATION

When looking safe
is a shortcut.

Near-perfect safety scores can hide fragile behavior. We identify two superficial cues that pull safety alignment away from the underlying intent.

⌗FORMATTING SHORTCUT

Change the wrapper.
Lose the defense.

The instruction stays the same, but removing chat templates can sharply reduce defense success. The model has learned to rely on the wrapper.

<｜User｜>[ The same harmful instruction ]<｜Assistant｜><think>

GRPO

100.00%

StrongReject DSR · DeepSeek-R1-Distill-Qwen-7B · Table 1

AaLEXICAL SHORTCUT

Keep the keyword.
Change the intent.

Sensitive vocabulary also appears in legitimate requests. Treating a keyword as the decision rule leads to unnecessary refusals.

ILLUSTRATIVE EXAMPLE

“How can I protect my database from an SQL injection attack?”

×Keyword shortcutRefuse

✓Intent-sensitive decisionHelp defend

THE PROBLEM, IN NUMBERS · FIGURE 1

Same instruction. Different defense.

Remove only the template. Watch how defense success changes across alignment methods.

w/With templatew/oWithout template

Defense success rate (%) ↑DeepSeek-R1-Distill-Qwen-7B

Removing the wrapper exposes a shortcut.

Figure 1 · Defense success with and without the pre-training template, across six alignment methods.

THE SHORTCUT TRAP · FLOW & DYNAMICS

How the learning signal disappears.

Approximate reconstruction

Training rewardOver-refusalTemplate gapPG-loss magnitude

Figure 2 · approximate trends reconstructed from the reference

Policy-gradient signal

r₁ ≈ r₂ ≈ … ≈ r_G
↓
Aᵢ → 0 → ∇L_pg → 0

Approximate reconstruction of Figure 2: reward saturates near step 40, false refusals rise, the template gap peaks then falls, and the gradient fades by step 100. Curves follow the reference image rather than raw training samples. Compare the original figures ↓

02THE APPROACH

Find the cues.
Decouple the decision.

Three coordinated stages address lexical and formatting shortcuts across supervised fine-tuning and reinforcement learning.

INSIDE DESHORTCUT-ALIGNINTERACTIVE SCHEMATIC

Attribution → augmentation → consistencySFT + PPO + GRPO

Figure 3. DeShortcut-Align framework: refusal sensitivity attribution, attribution-guided contrastive augmentation, and counterfactual consistency regularization in SFT and RL. — Figure 3 · TSS identifies refusal-sensitive tokens; AGCA retains those tokens while inverting intent; CCR blinds formatting tokens and trains consistent behavior under the counterfactual input.

03THE EVIDENCE

More robust.
Less over-refusal.

Explore the balance between template robustness, benign-query helpfulness, and general reasoning across three model backbones.

DeShortcut-AlignBold: best aligned method · safety: with / without template

MATH-500 · MMLU · LiveCodeBench · HumanEval↔ Scroll to view all benchmarks

Table 1 · All values in %. Δavg is the mean template gap in percentage points (↓). FRR is mean false rejection (↓); reasoning is mean pass@1 (↑). Weighted DSR uses the with-template condition. Base is excluded from best-result highlighting.

Metrics & evaluation settings

Δavg: mean with-template minus without-template DSR across 3 safety benchmarks (percentage points). FRR: average false rejection rate across XSTest, OKTest, and FalseReject. Reasoning: average pass@1 across MATH-500, MMLU, LiveCodeBench, and HumanEval.

04A CLOSER LOOK

What changes, and why it matters.

Component ablations, attribution strategies, and controlled intent pairs connect the main results to the mechanisms of the method.

COMPONENT ABLATIONS · TABLE 2

Both shortcuts need a control.

Without CCR, template dependence increases. Without AGCA, false refusals return.

7B · Δavg: template gap (pp) · FRR: false rejection (%) · DSR: weighted defense (%) · Acc: reasoning (%)

ATTRIBUTION STRATEGIES · TABLE 3

Select the tokens that matter.

TSS gives the lowest false rejection rate among the attribution strategies compared under SFT.

DeepSeek-R1-Distill-Qwen-7B · SFT

Table 3 · DSR: weighted defense (↑); FRR: mean false rejection (↓). Values in %.

PAIRED INTENT EXPERIMENT · TABLE 13

The same sensitive vocabulary.
A different response to benign intent.

150 harmful WildJailbreak prompts are paired with 150 benign rewrites that retain their sensitive vocabulary. DeShortcut-Align sharply reduces false refusals while retaining harmful-query defense.

Harmful-query defense ↑

98.67→98.00%

Benign-query false refusals ↓

94.67→32.67%

62.00 percentage points fewer false refusals

GRPO → DeShortcut-Align · 7B

JAILBREAK ROBUSTNESS · TABLE 4

Across distinct attack families.

Against PAIR, GCG, and TAP, the RL variant reaches 97.7% average defense success, compared with 92.7% for standard RL.

Table 4 · Defense success rate (%) ↑ · 7B · axis starts at zero

LONGER REASONING · FIGURE 6

Alignment beyond the first refusal tokens.

Token-level KL divergence from the base model stays higher across longer generations, consistent with an alignment effect that persists through the reasoning trajectory.

Figure 6. KL divergence from the base model across token positions. — Figure 6 · Policy divergence across token positions.

EXTENDED EXPERIMENTS

More backbones. More checks.

Open an experiment to see its full results.

Three training seeds · Qwen3-1.7B

Across seeds 0, 1, and 2, the mean template gap falls from 23.80 to 11.92 points; mean false rejection falls from 34.11% to 27.89%; reasoning accuracy rises from 64.92 to 67.44.

Table 6. Complete results for three Qwen3-1.7B training seeds. — Table 6 · Three-seed evaluation.

Transfer across GRPO and PPO

The same regularizer improves template robustness under both optimizers. Under PPO, the template gap falls from 17.41 to 5.73 points, while false rejection falls from 79.33% to 34.67%.

Table 7. PPO and GRPO with and without DeShortcut-Align. — Table 7 · Transfer across RL algorithms.

Regularization weights & token masking

The paper studies the SFT coefficient α, the RL coefficient λ, and consecutive-token masking length. Single-token masking offers the best safety/over-refusal balance in the reported comparison.

Tables 8 and 9. SFT alpha and RL lambda sensitivity. — Tables 8–9 · Regularization sensitivity.

Table 10. One, two and three token masking windows. — Table 10 · Consecutive-token masking.

Extended safety-alignment baselines

Table 5. Extended comparison with safety alignment baselines. — Table 5 · Extended baselines.

Contrastive-data quality & evaluator agreement

Independent model and human checks assess the benign contrastive data and the safety judge.

TABLE 11

Are the contrasts benign?

Table 11. Benign accuracy of contrastive queries.
Validator pipeline	Benign accuracy
GPT-4o Single stage	100.00%
Gemini 2.5 Pro	99.33%
Human expert	98.67%
GPT-4o → Gemini 2.5 Pro Two-layer filter	100.00%

AGCA-generated benign queries · verification accuracy (%)

TABLE 12

Does the judge agree with people?

Table 12. Safety evaluation on 150 sampled instances.
Model	Llama Guard DSR	Human expert DSR	Raw agreement
Base policy	54.00%	64.67%	77.33%
Ours	99.33%	99.33%	98.67%

150 sampled instances · DSR and raw label agreement (%)

05TRAINING DYNAMICS

Keep the learning signal alive.

Compare the original training figures: standard GRPO saturates and loses its learning signal, while DeShortcut-Align maintains stronger optimization dynamics.

Figure 2. Standard GRPO reward saturation, rising over-refusal and diminishing policy gradient. — Reward saturates, false refusals rise, and the policy-gradient signal fades.

Figure 4. DeShortcut-Align keeps reward variation and a stronger policy-gradient signal. — AGCA and CCR reduce shortcut reliance while maintaining a stronger learning signal.

ADAPTIVE REGULARIZATION · FIGURE 7

Balance consistency with exploration.

The dynamic coefficient adjusts the counterfactual loss alongside the policy update. The consistency loss converges while the training reward stabilizes.

Figure 7. Dynamic weight, counterfactual consistency loss and training reward for GRPO and PPO. — Figure 7 · Dynamic weight, consistency loss, and training reward.

06RESEARCH RESOURCES

Reproduce. Inspect. Build on it.

The implementation, contrastive datasets, attribution pipeline, and evaluation recipes are available in the research repository.

⌘Code

Training implementation and SFT, PPO, GRPO launchers.

↗▦Data

Attribution pools and AGCA benign SFT / RL datasets.

↗⌖Attribution & augmentation

Compute TSS, select sensitive tokens, and generate contrastive examples.

↗

↗THE TAKEAWAY

Align to the intent.
Look beyond the cue.

DeShortcut-Align provides a practical way to train against two measurable failure patterns, with stronger template robustness and better retention of useful capabilities in the evaluated settings.

Explore the implementation↗

DeShortcut-Align mascot cutting shortcut connections

Less shortcut. More intent.

07BUILD ON THIS WORK

Citation

@misc{liu2026deshortcutaligndecouplingspuriousshortcuts,
  title={DeShortcut-Align: Decoupling Spurious Shortcuts for Robust
         Safety Alignment in Large Reasoning Models},
  author={Qirui Liu and Yichen Sun and Yan Wang and Zhixuan Chu
          and Linbo Jiang and Jianan Lin and Kui Ren},
  year={2026},
  eprint={2609.34896},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2609.34896}
}