From Weeks to Hours: Fast and Principled
SFT Data Generation for Reasoning LLMs

Hongyi Henry Jin1 Wenhan Yang1 Meysam Ghaffari2 Carlos Morato2 Baharan Mirzasoleiman1
1University of California, Los Angeles 2Optum AI
Paper Code Data
Figure 1. Math reasoning accuracy and curation compute for OpenThoughts with 89k examples and SFTPROBE with 5k and 7.5k examples. Curation compute is 176, 3.3, and 4.3 H100 GPU-days, respectively.
Figure 1. Comparison with OpenThoughts-Math. Training on 5k curated examples matches performance with 53× speedup; 7.5k examples improve accuracy by 2.5% with 41× speedup. Curation compute is reported in H100 GPU-days.

Abstract

Reasoning-oriented post-training enables large language models (LLMs) to solve complex tasks via multi-step inference. Supervised fine-tuning (SFT) on high-quality reasoning traces is a particularly efficient approach, but its effectiveness depends critically on careful data curation. Existing pipelines rely on a costly generate-then-filter paradigm: large pools of long reasoning traces are produced by multiple teacher models and subsequently filtered for difficulty and diversity using additional LLMs, often requiring weeks of computation. In this work, we propose SFTPROBE, a fundamentally different approach that bypasses this pipeline by directly predicting the quality of reasoning data. We show that a brief LoRA-based adaptation, combined with evaluating loss on only 5% of each reasoning trace, suffices to estimate both difficulty and diversity. SFTPROBE is grounded in a geometric view of post-training, where pretrained LLMs lie in a low-rank, anisotropic loss basin. By probing this basin via low-rank perturbations, we construct compact loss signatures whose clustering captures diversity, while loss at curvature inflection points provides a robust proxy for difficulty. We also provide a simple criterion for teacher selection. Empirically, our approach reduces data curation time from weeks to hours while maintaining or improving performance. Remarkably, on mathematical reasoning, 5k generated examples match the highly filtered 89k-example in OpenThoughts-Math dataset, and 7.5k generated examples outperforms it by 2.5% in accuracy. We further introduce a high-quality physics reasoning dataset, and use it to train a model that outperforms an IPhO 2025 silver-medal-winning model.

Efficient SFT Data Curation

Predicting Similarity of Reasoning Traces

During training, each example updates the model parameters through its gradient. Examples with similar gradients induce similar parameter updates and therefore provide largely redundant training signals.

Our key observation is that fine-tuning occurs within a low-rank, relatively flat, and smooth region of the loss landscape, where the loss around the pretrained initialization is well approximated by a near-parabolic geometry. Specifically, we show that loss difference at the pretrained parameters and boundaries of the parabolic basin—corresponding to inflection points where the curvature changes—induces the tightest bound on gradient similarity throughout fine-tuning:

Theorem 3.1 If two examples have similar loss at the extremities of the basin, then their optimization dynamics remain similar across the entire training process. Formal statement

Theorem 3.1

Assume a smooth fine-tuning loss with bounded curvature \(0 \leq c \leq c_H\). Starting from initialization \(\theta_0\), for any checkpoint \(\theta \in \Theta\) during training define \(\epsilon_\theta = \|\theta-\theta_0\|_2\). Assume that fine-tuning gradients are predominantly captured by an \(r\)-dimensional subspace \(V \subset \mathbb{R}^d\) with orthonormal basis \(\{v_1,\ldots,v_r\}\), such that for every training example \(z\), \(\|\operatorname{Proj}_{V^\perp}\nabla L_z(\theta)\| \leq \gamma\). For each basis direction \(v_i\), denote \(c_{H,i} \triangleq c_{H,v_i}\), and let

\[ \theta_{i,1}=\theta_0+\epsilon_{i,1}v_i,\qquad \theta_{i,2}=\theta_0+\epsilon_{i,2}v_i,\qquad 0\leq\epsilon_{i,1}<\epsilon_{i,2}. \]

If any pair of training examples \(z_1,z_2\) have bounded loss difference at all checkpoints \(\theta_{i,1},\theta_{i,2}\):

\[ |L_{z_1}(\theta_{i,1})-L_{z_2}(\theta_{i,1})|\leq\delta,\qquad |L_{z_1}(\theta_{i,2})-L_{z_2}(\theta_{i,2})|\leq\delta, \quad \forall i\in\{1,\ldots,r\}, \]

then for any \(\theta\in\Theta\) during training, the (full-rank) gradient difference is bounded by

\[ \|\nabla L_{z_1}(\theta)-\nabla L_{z_2}(\theta)\| \leq \sqrt{\sum_{i=1}^{r}\left( \frac{2\delta}{\epsilon_{i,2}-\epsilon_{i,1}} +\frac{1}{2}(\epsilon_{i,1}+\epsilon_{i,2})c_{H,i} +2\epsilon_\theta\sqrt{c_Hc_{H,i}} \right)^2}+2\gamma. \]

To probe the basin, we briefly train a LoRA adapter on a very small subset of reasoning traces to identify its dominant directions. However, the loss landscape is anisotropic: curvature varies substantially across directions, implying that the extent of the near-parabolic region differs from one direction to another.

Figure 2(b–c). LoRA directions have different curvatures. Directions with lower curvature should be perturbed more to reach the inflection point.
Figure 2(b–c). LoRA directions have different curvatures. Directions with lower curvature should be perturbed more to reach the inflection point.

We perturb the model incrementally along each direction and fit a curve to the resulting loss profile to identify the inflection point where curvature changes sign (Algorithm 1). This gives a boundary checkpoint for each direction.

We evaluate the loss of only the initial 1k reasoning tokens at these perturbed checkpoints to construct a compact loss signature for each example. We then cluster these signatures and select an equal number of examples from each cluster to obtain a diverse subset.

Predicting Reliable Difficulty Scores

When generating reasoning traces, the teacher model starts its reasoning by summarizing and rephrasing the problem. The loss of the initial 100 tokens provides a high-quality signal for problem difficulty.

Harder problems have higher curvature in this loss. While loss at the pretrained model does not correlate well with difficulty, perturbing toward the inflection point separates the difficulty levels. We therefore evaluate this loss at an inflection checkpoint along a random direction to obtain a difficulty score, without training on full reasoning traces.

We discard the easier cluster of problems within each source and sample more examples from sources with higher average difficulty. We then generate full reasoning traces for the selected examples.

Teacher Selection

We aim to select the teacher whose language is most familiar to the student, allowing the student to focus on solving the problem rather than parsing unfamiliar phrasing. To this end, we randomly sample a set of questions and prompt each candidate teacher model to rephrase the same problems. We cap each response at 100 tokens, as in difficulty filtering, and evaluate the rephrasings using the pretrained student parameters rather than the perturbed ones. As shown in Figure 4, the teachers with the lowest average loss can best train the student model.

Figure 4. We evaluate the loss of the student model and the accuracy of the finetuned on the dataset generated from the teacher model on the NuminaMath dataset.
Figure 4. We also train the student using the same randomly selected 1000 questions with responses generated from the teacher model. We find that the higher the rephrasing loss is, the worse the final finetuned student performs.

Main Results

Data Selection for Medical Reasoning

Figure 5. Medical reasoning accuracy versus selection time for Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct, comparing SFTPROBE with selection baselines.
Figure 5. Selecting 1k examples from the pre-generated M23k dataset. Our method is comparable to the best selection baseline on Qwen2.5-7B and outperforms selection baselines on Llama3.1-8B, while being substantially faster. Wall time on 4×H100 is reported.

Data Curation for Medical Reasoning

Figure 6. Medical reasoning accuracy versus curation time. SFTPROBE with 1k generated examples closes 76 percent of the gap between random sampling and full 23k-example training at 7 percent of the compute cost.
Figure 6. Compared with full 23k-example fine-tuning, our curated 1k examples close 76% of the gap between random sampling and full-dataset training, while requiring 7% of the compute cost. Wall time on 4×H100 is reported.

Data Curation for Mathematical Reasoning

Figure 7. Math reasoning accuracy versus curation time for 1k, 5k, and 7.5k SFTPROBE examples and selection baselines using OpenThoughts-Math.
Figure 7. Data selection baselines select 1k examples from pre-generated OpenThoughts-Math traces. Our method generates data with QwQ-32B and takes less time even when generation time is included. Wall time on 4×H100 is reported.

Data Curation for Physics Reasoning

Table 1. Experiment results on data curation for physics reasoning with Qwen2.5-7B-Instruct as the student model. We outperform PRIME-RL/P1-30B-A3B, a model capable of winning the silver medal for IPhO 2025, although we start with a much weaker base model.
ModelAverage score
Qwen2.5-7B-Instruct3.58
Google/Gemma-4-26B-A4B-it9.65
PRIME-RL/P1-30B-A3B9.05
Ours (1k data generated)9.14 ± 0.29
Ours (2k data generated)9.32 ± 0.11
Ours (5k data generated)9.76 ± 0.74

BibTeX

@inproceedings{anonymous2026from,
  title={From Weeks to Hours: Fast and Principled {SFT} Curation for {LLM}},
  author={Anonymous},
  booktitle={The Fortieth Annual Conference on Neural Information Processing Systems},
  year={2026},
  url={https://openreview.net/forum?id=4jOHUJJVCg}
}