Condensing Chain-of-Thought

Overview

Reasoning models show their work, but much of it is filler: repeated steps and vague "fluffy" language that cost compute without making the answer more likely to be right. We tested whether training a small model on condensed reasoning would make it a better mathematician.

Method

• Solved 137 problems from GSM8K, past AIMEs and the Putnam with DeepSeek-R1-Distill-Qwen-14B, keeping only problems the base model got right
• Split each solution into its reasoning and its conclusion, and rewrote the reasoning with Qwen2.5-32B-Instruct (3-bit) to cut redundancy without changing the answer
• Fine-tuned DeepSeek-R1-Distill-Qwen-7B on the condensed solutions with LoRA for four epochs, on a single RTX 4060 Ti
• Split condensed outputs longer than the context window into overlapping segments to keep the logic continuous

Results

The fine-tuned model beat the base model on every exam we tried, including AIME 2024, which it never saw in training.

• AIME 2024, one attempt: 53.3 vs 46.7, up 14.1% (the average human score is 40)
• AIME 2024, three attempts: 66.6 vs 56.6, up 17.7%
• AMATYC Student Math League, Fall and Spring combined: up 15.8%

Why it matters

Fine-tuning a small model on concise reasoning made it more accurate without heavier hardware. That points to cheaper, clearer reasoning models for tutoring, proof assistance and debugging. [What you'd test next with more data or full precision.]