🔍 Read the full analysis: Fine-Tuning Takes More Time Than FLOPs Alone Predict on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Hugging Face reports that a CatBoost model predicted the duration of 272 production supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The best tested FLOP-based estimate had a 12.41-hour error, but the study covers one production environment and has not yet been published in full.
Hugging Face says a CatBoost model predicted the end-to-end duration of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation, compared with 12.41 hours for the best tested FLOP-based estimate, as detailed in the original analysis. The result could help GPU operators schedule jobs, but it is based on one production environment and does not yet show that using the predictor reduces delays or idle GPU time.
The runs used shared NVIDIA H200 GPUs and accounted for about 7,382 GPU-hours. Individual jobs lasted from 1.6 to 172.4 hours. The dataset included model families up to 15 billion parameters, both LoRA and full fine-tuning, different dataset sizes, and jobs assigned one, two or four GPUs. The broader pool comprised 8 to 32 H200s across one to four nodes, with other training and inference workloads running at the same time.
Hugging Face defines runtime as wall-clock time from GPU allocation to job completion; queue time is excluded. The model used 24 features, including configuration and model metadata available before dispatch, such as evaluation cadence, checkpoint settings, per-GPU token volumes and fine-tuning method. A FLOP-based duration estimate was itself included as one feature.
In stratified five-fold cross-validation, the model recorded a 1.71-hour mean absolute error, a 0.51-hour median error, an R² of 0.98 and a 3.52-hour root mean squared error. The FLOP baseline at 30% Model FLOPs Utilization (MFU) had a 12.41-hour mean absolute error and an R² of 0.31. Its error was 13.01 hours at 45% MFU and 14.62 hours at 60%. The preview says testing MFU assumptions from 25% to 65% did not resolve the baseline’s shortcomings.
Why Fine-Tuning Takes Longer Than FLOPs Say
Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.
What the estimates capture
The target is time from GPU allocation to job completion; queue time is excluded.
The training loop
FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.
The whole job
CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.
Other runtime costs
Setup, data handling, network and storage can add time alongside training.
Prediction error in the tested data
Lower error means predicted duration was closer to observed duration on average.
What this could mean for GPU scheduling
More useful duration estimates could help teams plan scarce GPU capacity.
- GPUs may sit idle until another job starts.
- Operators can misjudge when capacity will be free.
- Other work can be delayed.
- Delivery dates may shift.
How far the result reaches
The preview reports cross-validation on one corporate production environment.
Runs used shared NVIDIA H200 GPUs and varied configurations.
Accuracy on other clusters, software stacks and workload mixes is not established.
The preview reports prediction error, not savings in GPU time or scheduling delays.
Better Estimates Could Aid GPU Scheduling
Runtime predictions help cluster operators decide when to dispatch work and how to allocate limited accelerators. If a job finishes sooner than expected, GPUs may sit idle until another job starts; if it runs long, it can delay other work and shift delivery schedules. A more accurate estimate could help teams plan allocations around those risks, especially on shared clusters where jobs compete for resources.
The reported error gap is substantial within this dataset, but prediction accuracy is not the same as demonstrated operational savings. The preview reports model performance, not a controlled scheduler deployment. It does not quantify whether the approach reduced GPU-hours, costs, idle intervals or delivery delays. Those outcomes would need to be measured separately.
Products Worth Considering

DJI Mini 5 Pro Fly More Combo with DJI RC 2, 1-Inch CMOS

DJI Mini 4K Drone Combo with 4K Camera, 3-Axis Gimbal, and Two Batteries

Samsung 32 Odyssey G5 QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
Why FLOPs Miss Parts of Runtime
A common training estimate approximates dense Transformer compute as six times the parameter count and training-token count, then divides by GPU count, peak throughput and an assumed MFU. MFU represents the share of theoretical peak throughput achieved in practice; the preview describes a typical range of roughly 30% to 60%. This calculation focuses on training compute and depends on assumptions about hardware utilization.
The study’s target includes more than the training loop. Setup, model loading, tokenization, scheduled evaluations, checkpoint uploads and final storage can all add wall-clock time. Hugging Face says these costs vary with configuration and can compete for network and storage capacity in a multi-tenant environment. The researchers included scheduled evaluation tokens in the baseline and tested several MFU values rather than relying on one assumption.
The preview says the team compared several candidate approaches before selecting CatBoost and tuning its features. It describes the records as production runs, but the evaluation was cross-validation within the same dataset, not a test across independently collected clusters. The announced full paper is not yet available.
“Everything is known before dispatch.”
— Hugging Face study preview
How Well Results Travel
The preview does not establish whether the reported accuracy will hold on other GPU types, clusters, software stacks or workload mixes. Its 272 jobs came from one corporate production environment, and cross-validation on those records does not substitute for testing on a separately collected site or later time period.
The full paper may provide more information about the dataset, evaluation design and feature selection. The preview does not report a live scheduling trial or enough detail to judge how performance changes as workloads and cluster conditions shift. It is also unclear how often the predictor would need retraining. The reported scores are evidence of prediction performance on this dataset, not proof of a general-purpose runtime advantage.
Full Paper and External Testing
Hugging Face says it plans to post the full version on arXiv, but has not provided a publication date. Further details in that paper could help readers assess the dataset and validation methods. Independent testing across different periods or environments would show whether the prediction gap persists beyond the production pool used here.
A subsequent operational trial could test whether feeding estimates into a scheduler leads to fewer delays or idle GPU intervals. Until those results are available, the practical scheduling benefit remains unmeasured, and the study’s results should be read as a promising finding from a single environment rather than a settled result across GPU clusters.
Key Questions
What did Hugging Face report?
Its study preview says a CatBoost model predicted runtimes for 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation.
How did the FLOP-based estimate perform?
The best tested setting, at 30% MFU, had a 12.41-hour mean absolute error. The preview also reports errors of 13.01 hours at 45% MFU and 14.62 hours at 60% MFU.
What does the runtime measurement include?
It measures wall-clock time from when a job receives GPUs until it finishes. Queue time is excluded; setup and other job-lifecycle overheads can contribute to the measured duration.
Does the result prove the model will work on other clusters?
No. The jobs came from one corporate production environment, and the reported cross-validation does not establish accuracy on other hardware, software stacks or independently collected workloads.
Has the predictor been shown to save GPU time?
The preview reports prediction errors, not a controlled deployment’s effects. It does not quantify GPU time, cost, idle intervals or delays saved by using the model in a scheduler.
Primary source: Hugging Face · via ThorstenMeyerAI.com
