Roads to Rome

Token-level routing between small and large reasoning models. R2R learns which tokens need the large model, leaving most generation to the small one.

2.8×
Generation speed vs R1-32B
5.6B
Avg. Activated Params
1.6×
Avg. accuracy vs R1-7B
Read Paper View Code 🤗 Models NeurIPS 2025

The Challenge

LLMs are powerful but expensive.

Small models are fast but make mistakes. What if we could get the best of both worlds?

🚀

Small Models (1.5B)

Fast generation
Limited accuracy

←
⚡

R2R Routing

Best of both worlds
Smart token routing

→
🧠

Large Models (32B)

Accurate reasoning
Slow & expensive

The Discovery

Not all tokens are equal.

In the 7.6M-token routing training set, R1-1.5B and R1-32B agree on most next-token predictions under the same context. Only about 5% of tokens are labeled path-divergent.

89%
Identical Tokens
Both models predict the same next token under the same context, so the small model’s output can be used.
6%
Neutral Differences
Minor variations like "let's" vs "let us" that don't affect reasoning outcomes.
5%
Divergent Tokens
Differences that change the reasoning path. The router learns to identify these tokens for correction by the LLM.

Identical tokens occur when both the small and large language model predict the exact same next token given the same context. Since they agree, we can safely use the faster small model.

Neutral differences are different predictions whose sentence-level continuations remain equivalent in meaning, such as abbreviations or stylistic variations.

Divergent tokens genuinely alter the meaning, logic, or conclusion of the current sentence, thus diverging the subsequent reasoning path. These are the critical tokens where we need the large model's guidance.

Interactive Demo

Watch R2R in action.

An animated example of token-level routing. The preset continuations illustrate offline labeling, rather than live model output.

R2R Inference Workflow
Step 1
Input Query
Step 2
SLM Predicts
Step 3
Router Checks
Step 4
Route Decision
Step 5
Output Token
AIME 2024 Problem I-1
Every morning Aya goes for a 9-kilometer-long walk and stops at a coffee shop afterwards. When she walks at a constant speed of s kilometers per hour, the entire walk takes exactly 4 hours, including t minutes spent at the coffee shop. When she walks at a constant speed of s + 2 kilometers per hour, the entire walk takes exactly 2 hours and 24 minutes, including t minutes spent at the coffee shop. Suppose Aya walks at s + ½ kilometers per hour. Find the number of minutes the entire walk takes, including the t minutes spent at the coffee shop.
Click the play button above to see R2R process this problem
Small Model (R1-1.5B) generates tokens...
🔍
Neural Router Analyzing Token
The 56M-parameter neural router examines three features from the SLM: top-100 logits (prediction uncertainty), token embeddings (frequency bias), and hidden states (semantic context) to predict if this token would diverge the reasoning path.
✓
Accept SLM Token
Identical or neutral
⚡
Route to LLM
Divergent token detected
Generated Response with R2R Routing
SLM Token (Identical)
SLM Token (Neutral)
LLM Token (Divergent)

Continuation comparisons illustrate the neutral and divergent labels.

Watch Video Demo

Side-by-side comparison: R1-32B (left) vs R2R (right). Red tokens are routed to LLM.

How It Works

Intelligent token routing.

R2R uses a lightweight neural router trained on automatically generated token-level labels to decide which tokens need the large model's attention.

R2R Method Overview
Automatic Data Generation Pipeline

A key contribution of R2R is an automatic pipeline for generating token-level routing labels without expensive human annotation:

  1. LLM generates the reference response.
  2. SLM prefills that response to find differing next-token predictions.
  3. LLM continues each differing SLM token to the sentence boundary; the reference continuation comes from the original response.
  4. Verifier model compares the sentence continuations to label neutral or divergent differences.

This pipeline produces large-scale training data automatically, enabling the router to learn which token differences actually matter for reasoning.

Data Generation Pipeline
Path-Following Routing Strategy

When generating routing labels, we compare the next-token predictions of the SLM and LLM. Identical predictions can use the SLM output. For differing predictions, continuation and verification determine whether the difference is:

  • Neutral: Doesn't affect reasoning outcomes (use SLM)
  • Divergent: Alters the reasoning path (route to LLM)

Sentence-level verification provides offline training labels. During inference, the neural router predicts divergence from SLM features and invokes the LLM when its score exceeds a threshold. It approximates the labeling strategy rather than verifying every continuation.

Neural Router Architecture

The router is a lightweight 56M-parameter feed-forward network that takes three inputs from the SLM:

  • Top-100 SLM logits: Captures prediction uncertainty (higher entropy indicates potential divergence)
  • Token embeddings: Encodes token frequency biases (rare tokens are more likely to diverge)
  • Last-layer hidden states: Provides semantic context for the current generation step

A divergence score enables immediate routing decisions. Adjusting the threshold changes LLM usage without retraining; the reported configuration uses the LLM for 11–15% of tokens.

Predictive Indicators of Divergence
Entropy Distribution

SLM Entropy

Average SLM output entropy is 3.8× as high for divergent tokens as for non-divergent tokens, indicating greater prediction uncertainty.

Token Frequency

Token Frequency

Low-frequency tokens are more likely to be divergent in the training data, suggesting token rarity is a useful predictive signal.

Performance

Higher accuracy at lower inference cost.

R1-1.5B + R1-32B reaches 46% average accuracy with 5.6B average activated parameters, versus 43% for R1-14B and 50% for R1-32B.

R1-7B (Baseline)
28%
Average Accuracy
7.0B params
154K cost*
R1-14B
43%
Average Accuracy
14B params
234K cost*
R2R (Ours)
46%
Average Accuracy
5.6B avg params
103K cost*
R1-32B (LLM)
50%
Average Accuracy
32B params
537K cost*

*Cost is a decoding-cost proxy: output token count × average activated parameters (billions), including the SLM, router and weighted LLM usage. It is not measured latency.

On AIME with two A800-80GB GPUs, generation speed rises from 30.5 to 84.3 tokens/s (2.76×); mean request latency falls from 498 to 218 seconds (2.28×). See Table 3.

Pareto Frontier Chart

R2R consistently advances the Pareto frontier across AIME, GPQA, and LiveCodeBench benchmarks.

Detailed Benchmark Results

Table 2 setup: AIME 2024–2025, LiveCodeBench (Aug 2024–Jan 2025), and GPQA-Diamond; greedy decoding with a 32K output limit. The last column averages activated parameters across benchmarks.

Method AIME LiveCodeBench GPQA Avg. Params
R1-1.5B (SLM) 12% 9% 8% 1.5B
R1-7B 32% 24% 29% 7.0B
R1-14B 48% 38% 44% 14.0B
R2R (Ours) 55% 39% 44% 5.6B
R1-32B (LLM) 57% 45% 46% 32.0B
Routing Behavior Analysis
Routing Behavior

R2R naturally learns efficient routing patterns: fewer LLM calls during straightforward reply phases, and more at the beginning and end of each thought where reasoning direction is determined.

Team

Meet the researchers.

Tianyu Fu1,2* Yi Ge1* Yichen You1 Enshu Liu1 Zhihang Yuan2 Guohao Dai3,2 Shengen Yan2 Huazhong Yang1 Yu Wang1
1Tsinghua University    2Infinigence AI    3Shanghai Jiao Tong University
*Equal contribution

Citation

Cite our work.

If you find R2R useful for your research, please consider citing our paper.

BibTeX 
@article{fu2025r2r,
          title={R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing},
          author={Fu, Tianyu and Ge, Yi and You, Yichen and Liu, Enshu and Yuan, Zhihang and Dai, Guohao and Yan, Shengen and Yang, Huazhong and Wang, Yu},
          journal={arXiv preprint arXiv:2505.21600},
          year={2025}
        }