Learning from experience. One critical step at a time.

Selective Critical-Step
On-Policy Self-Distillationfor Agentic Reinforcement Learning

Qirui Liu^1,*, Wei Cao^1,*, Yu Mi¹, Yichen Sun¹, Xuekun Zhang¹, Lei Liu², Yue Shen², Zhixuan Chu^1,†

¹ Zhejiang University² Ant Group

* Equal contribution† Corresponding authorzhixuanchu@zju.edu.cn

Explore the method Code Models & data Training curves Read our paper

Better agents learn where it matters. SCOPE turns self-generated hindsight into targeted corrections at critical error steps.

Qwen2.5-3B · Main results at training step 150

ALFWorld success

97.7%

+5.9 pp over SEED

WebShop success

85.2%

+6.3 pp over SEED

See how SCOPE learns

01The motivation

An outcome tells only part of the story.

A successful trajectory can contain mistakes. A failed one can contain good decisions. Corrective guidance needs to match the step that actually needs it.

01 / OUTCOME ≠ STEP QUALITY

Success ≠ every step is right

An agent can recover from mistakes and succeed. A failed trajectory can still contain good decisions.

Trajectory A✓Success

Trajectory B×Failure

Correct stepError step

29.8%of steps in successes are errors

42.4%of steps in failures are correct

02 / TRAJECTORY-LEVEL SKILL, OFF-TARGET SIGNAL

A skill ≠ guidance for every step

One hindsight skill conditions the teacher for an entire trajectory, including decisions unrelated to that skill.

SKILL-CONDITIONED TEACHER“Check the product constraints.”

One trajectory-level skill → every step

Skill-relatedSkill-unrelated

57.9%of nonzero sampled-token gaps occur at skill-unrelated steps

SCOPE motivation and main results. — Figure 1 · Motivation and results. Outcome–step mismatch, off-target skill signals, training dynamics, and Qwen2.5-3B main results.

02The approach

Find the error. Focus the learning.

The same policy acts, analyzes its own experience, and learns from step-specific skills. Auxiliary distillation is applied only at selected error steps.

THE SCOPE LEARNING LOOP

Explore the three ideas that turn hindsight into better decisions.

SELF-BOOTSTRAPPED ERROR ANALYSIS

A skill for the step that needs it.

The analyzer reads the trajectory, grounds each error in feedback, and writes a correction tied to its target steps.

RECORDED ON-POLICY TRAJECTORY

TASK

AnalyzingAnalyzer-selected stepOrdinary RL only

SAME POLICY SNAPSHOT

Analyzer · A_θ̄

STRUCTURED STEP-LEVEL GUIDANCE0 skills

Auxiliary selection mask mₜ

Each skill conditions the teacher only at its assigned steps.

Analysis progress00 / 08

Our full method diagram

SCOPE method diagram. — Figure 2 · SCOPE: supervised analyzer initialization, self-bootstrapped critical-step analysis, selective bounded signed OPSD, and efficiency-aware GRPO.

01
Initialize the analyzer
A lightweight supervised warm-up teaches the policy to locate errors and write corrective skills. External annotation is confined to this offline stage.
02
Locate critical errors
During online RL, the policy analyzes both successful and failed rollouts, assigning each selected error step its own relevant hindsight skill.
03
Bound the correction
A bounded signed teacher–student coefficient preserves reinforcement and suppression while limiting extreme auxiliary update weights.
04
Reward efficient success
A success-relative pairwise length reward favors shorter successful trajectories within the same task group, without a fixed length target.

03Inside the trajectory

Real tasks. Every action, in view.

Follow recorded agents through an embodied room and an online store. Explore how each action changes the environment.

TaskPut a newspaper on the sofa.put a newspaper in sofa

Recorded trajectorySuccessful trajectory

3D reconstruction from a text trajectory

ALFWORLD

4 successful trajectories · 2 failed trajectories

Initial state

Room overview

Replay stepSpeed

Shopping task

Find me quick release, hands free iphone accessories for wireless charging with color: freemount, and style: dash, and price lower than 70.00 dollars

Recorded episodeSuccessful trajectory

WebShop / recorded environment

Loading recorded trajectories…

Replay stepSpeed

04The evidence

Stronger completion, from 1.7B to 32B.

SCOPE leads the smaller-backbone comparisons on task completion and scales to Qwen3-32B, improving ALFWorld success on both seen and unseen tasks.

THE RESULTS, AT A GLANCE

Better completion. Across backbones.

Explore our main results. Select a benchmark and toggle methods to compare.

Task success (%) ↑Qwen2.5-3B-Instruct

Table 1 · Our main results, matching the full comparison below. SCOPE uses training step 150. The y-axis begins at zero; bar values are percentages.

ALFWorld

SCOPE

Overall task success

97.7%

+5.9 ppvs. SEED

WebShop

SCOPE

Exact task success

85.2%

+6.3 ppvs. SEED

Full comparison · 9 methods

Scroll horizontally to see all metrics

BestSecond best

Table 1 · Our complete ALFWorld and WebShop results. Qwen2.5-3B-Instruct
Method	Guidance type	ALFWorldSuccess (%) ↑							WebShopAll values (%) ↑
Method	Guidance type	Pick	Look	Clean	Heat	Cool	Pick2	Overall	Score	Success
Vanilla	None	44.4	11.1	6.2	15.4	28.6	12.5	21.9	6.7	0.8
Skill-Prompt*	Skill bank	51.7	66.7	48.4	0.0	4.3	10.0	28.9	0.2	0.8
GRPO	None	91.2	62.5	96.2	61.9	65.0	47.4	75.0	79.8	63.3
AgentOPSD	Skill bank	100.0	68.8	86.4	66.7	84.6	73.7	84.4	90.4	69.5
SDAR	Skill bank	97.1	62.5	100.0	61.9	75.0	84.2	84.4	85.0	68.0
OPID	Hybrid skills	92.7	100.0	88.9	70.0	84.2	70.0	84.3	85.0	74.2
GRSD	Traj. skill	100.0	81.3	91.3	88.2	70.8	85.7	86.7	86.9	76.5
SEED	Traj. skill	100.0	100.0	100.0	100.0	70.6	80.0	91.8	88.5	78.9
SCOPE Ours	Step skills	100.0	100.0	100.0	100.0	85.7	100.0	97.7	87.9	85.2

Evaluation protocol & notes

All values are percentages. ALFWorld: success on six task families and the overall episode success rate (Overall), evaluated on 128 seen validation episodes. WebShop: mean normalized task score (Score) and exact task success (Success) on 128 fixed tasks. Type describes auxiliary guidance; hybrid skills combine trajectory- and step-level skills. SCOPE metrics are logged at training step 150. * Skill-Prompt receives skills at evaluation. Best and second-best distinct values are highlighted within each backbone, including ties.

LARGER-MODEL SCALING / QWEN3-32B

Near-perfect success at 32B, with fewer steps.

On ALFWorld, SCOPE improves over GRPO with the same Qwen3-32B backbone. The gains hold on both seen and unseen tasks.

Qwen3-32B

Seen success99.2%+14.0 pp vs. GRPO

Unseen success94.0%+12.7 pp vs. GRPO

Fewer seen-task interactions65.0%7.7 vs. 22.0 mean steps

Table 3 · Qwen3-32B results on ALFWorld
Split	Method	Pick	Look	Clean	Heat	Cool	Pick2	Overall	Mean steps ↓
Seen	GRPO	96.8	75.0	89.5	90.0	80.8	70.0	85.2	22.0
Seen	SCOPE	100.0	100.0	100.0	95.0	100.0	100.0	99.2	7.7
Unseen	GRPO	87.5	83.3	77.4	78.3	85.7	76.5	81.3	17.4
Unseen	SCOPE	83.3	100.0	93.5	91.3	100.0	100.0	94.0	9.2

Seen results: training step 150 on 128 validation episodes. Unseen results: 134 episodes. Success rates are percentages; mean steps count environment interactions per episode. Qwen3-32B training used 16 NVIDIA H200 GPUs.

Swipe to inspect all eight training curves ↔

Eight Qwen3-32B ALFWorld training curves comparing SCOPE and GRPO. — Figure 9 · Single-run seen-evaluation curves for six ALFWorld task families, overall success, and mean interactions. The dotted line marks training step 150.

APPENDIX B.2 / TRAINING DYNAMICS

Follow learning across both environments.

Single-run validation curves for Qwen2.5-3B and Qwen2.5-7B. Dotted lines mark training step 150, the checkpoint used in our main table.

Explore live logs on SwanLab

82.8%fewer RL steps to the ALFWorld success target

60.0%fewer RL steps to the WebShop success target

ALFWorld training curves for SCOPE-3B and SCOPE-7B. — Figure 10 · ALFWorld single-run validation: overall and task-family success, plus mean interactions. At step 150, success is 97.7% / 98.4% and mean interactions are 8.24 / 7.73 for 3B / 7B.

WebShop training curves for SCOPE-3B and SCOPE-7B. — Figure 11 · WebShop single-run validation: success, normalized task score, and mean interactions. At step 150, success is 85.2% / 84.4% and mean interactions are 6.02 / 5.95 for 3B / 7B.

05A closer look

Beyond the main benchmark.

Unseen tasks, optimization diagnostics, and component ablations examine what the complete system learns and how it trains.

ERROR-STEP LOCALIZATION

The analyzer learns where to intervene.

Supervised initialization improves error-step localization across both WebShop and ALFWorld. In our comparison, the 3B SFT analyzer can surpass the 7B base analyzer on micro-F1.

Error-step localization metrics for base and SFT analyzers on WebShop and ALFWorld. — Figure 7 · Error-step localization on WebShop and ALFWorld, comparing base and SFT analyzers at 3B and 7B.

01 / Generalization

More unseen tasks solved. Fewer interactions.

On unseen ALFWorld tasks, success rises from 66.4% with GRPO to 89.6% with SCOPE. Average interactions fall from 18.86 to 10.06.

89.6%unseen-task success

+23.2pp over GRPO

46.7%fewer interactions

Generalization on unseen ALFWorld tasks. — Figure 8 · ALFWorld unseen-task generalization. Success improves while average interactions decrease.

02 / Optimization

Keep extreme gaps in check.

We observe a sharp drop in student log-probability during minibatch optimization. Our bounded signed coefficient limits gap-driven amplification.

Teacher–student optimization diagnostics. — Figure 3 · Optimization anomaly: current student scores versus cached teacher scores. Dashed lines mark outer update 26.

03 / Interaction efficiency

Encourage shorter successful behavior.

Pairwise rewards compare interaction lengths among successes from the same task group. The plot reports mean environment interaction steps during ALFWorld training; lower is better.

Swipe to compare every curve ↔

Mean ALFWorld environment interaction steps during training for SCOPE and ablations. — Figure 5 · Mean environment interaction steps on ALFWorld. “All” denotes full SCOPE.

04 / Component ablations

Each component has a role.

At training step 180, full SCOPE reaches 98.4% ALFWorld success. Removing supervised initialization yields 92.2%; removing the length reward yields 93.0%.

ALFWorld component ablations and parameter sweeps. — Figure 4 · Qwen2.5-3B on ALFWorld: component ablations and parameter sweeps. Endpoint comparisons use step 180, separately from the step-150 main results.

Table 2 · ALFWorld component ablations · Qwen2.5-3B · training step 180
Variant	Pick	Look	Clean	Heat	Cool	Pick2	Overall
Full SCOPE	100.0	100.0	100.0	100.0	100.0	92.0	98.4
Initialization and guidance
w/o SFT initialization	100.0	92.9	100.0	70.6	83.3	94.7	92.2
w/o Step-Skill	100.0	92.9	100.0	94.1	94.4	89.5	96.1
Trajectory-level guidance	100.0	92.9	100.0	88.2	100.0	89.5	96.1
Skill-bank guidance	100.0	100.0	96.6	76.5	100.0	94.7	95.3
Length-reward design
w/o length reward	96.8	91.7	100.0	100.0	92.3	75.0	93.0
Absolute length bonus	100.0	100.0	100.0	94.1	100.0	89.5	97.7

Task success (%). Overall is micro-averaged across evaluation episodes. Best values, including ties, are bold.

05 / Parameter sensitivity

Three parameters. Clear trade-offs.

We vary α, c, and τₗₑₙ one at a time from the default setting (2.5, 1, 0.1). The three curves show overall ALFWorld success at training step 180.

Three parameter-sensitivity curves for gap slope, saturation bound, and pairwise temperature. — Figure 6 (left) · Overall ALFWorld success as α, c, and τₗₑₙ vary individually. The purple diamonds mark our default setting.

Purple diamond: our default · Blue circles: other settings

ALFWorld success at alpha values 1, 2.5, and 5. — α · 89.8% at 1; 98.4% at 2.5; 97.7% at 5.

ALFWorld success at saturation-bound values 0.1, 1, and 10. — c · 94.5% at 0.1; 98.4% at both 1 and 10.

ALFWorld success at pairwise-temperature values 0.05, 0.1, and 0.2. — τₗₑₙ · 93.8% at 0.05; 98.4% at 0.1; 96.1% at 0.2.

Swipe horizontally to see every task category ↔

Task-wise ALFWorld success heatmap for parameter settings. — Figure 6 (right) · Task-wise success for the same settings; Full denotes α = 2.5, c = 1, τₗₑₙ = 0.1.

06Research resources

Continue with the models, data, and runs.

Reproduce the analyzer warm-up with the released SFT models and datasets, then inspect the logged training runs.

SCOPE codeTraining and evaluation implementation SFT models & dataALFWorld and WebShop analyzers and supervised data SwanLab runsInteractive training curves and run details

Citation

If you use SCOPE in your research, please cite our paper.

arXiv:2610.38391 ↗

Selective Critical-Step
On-Policy Self-Distillationfor Agentic Reinforcement Learning

An outcome tells only part of the story.

Success ≠ every step is right

A skill ≠ guidance for every step