Fine-Tuning Takes More Time Than FLOPs Alone Predict
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Fine-Tuning Takes More Time Than FLOPs Alone Predict on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

Hugging Face reports that a CatBoost model predicted the duration of 272 production supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The best tested FLOP-based estimate had a 12.41-hour error, but the study covers one production environment and has not yet been published in full.

Hugging Face says a CatBoost model predicted the end-to-end duration of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation, compared with 12.41 hours for the best tested FLOP-based estimate, as detailed in the original analysis. The result could help GPU operators schedule jobs, but it is based on one production environment and does not yet show that using the predictor reduces delays or idle GPU time.

The runs used shared NVIDIA H200 GPUs and accounted for about 7,382 GPU-hours. Individual jobs lasted from 1.6 to 172.4 hours. The dataset included model families up to 15 billion parameters, both LoRA and full fine-tuning, different dataset sizes, and jobs assigned one, two or four GPUs. The broader pool comprised 8 to 32 H200s across one to four nodes, with other training and inference workloads running at the same time.

Hugging Face defines runtime as wall-clock time from GPU allocation to job completion; queue time is excluded. The model used 24 features, including configuration and model metadata available before dispatch, such as evaluation cadence, checkpoint settings, per-GPU token volumes and fine-tuning method. A FLOP-based duration estimate was itself included as one feature.

In stratified five-fold cross-validation, the model recorded a 1.71-hour mean absolute error, a 0.51-hour median error, an R² of 0.98 and a 3.52-hour root mean squared error. The FLOP baseline at 30% Model FLOPs Utilization (MFU) had a 12.41-hour mean absolute error and an R² of 0.31. Its error was 13.01 hours at 45% MFU and 14.62 hours at 60%. The preview says testing MFU assumptions from 25% to 65% did not resolve the baseline’s shortcomings.

At a glance
reportWhen: Study preview published; full paper for…
The developmentHugging Face’s study preview reports that a CatBoost model outperformed tested FLOP-based estimates when predicting the runtime of 272 fine-tuning jobs on shared H200 GPUs.
AI Training · Briefing

Why Fine-Tuning Takes Longer Than FLOPs Say

Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.

1.71 hours
CatBoost mean absolute error
12.41 hours
Best tested FLOP baseline error
272
Production jobs evaluated
7,382
Approximate GPU-hours

What the estimates capture

The target is time from GPU allocation to job completion; queue time is excluded.

Compute

The training loop

FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.

Configuration

The whole job

CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.

Shared systems

Other runtime costs

Setup, data handling, network and storage can add time alongside training.

Prediction error in the tested data

Lower error means predicted duration was closer to observed duration on average.

CatBoost1.71 hours
FLOP baseline, 30% MFU12.41 hours
FLOP baseline, 45% MFU13.01 hours
FLOP baseline, 60% MFU14.62 hours

What this could mean for GPU scheduling

More useful duration estimates could help teams plan scarce GPU capacity.

When jobs finish early
  • GPUs may sit idle until another job starts.
  • Operators can misjudge when capacity will be free.
When jobs run over
  • Other work can be delayed.
  • Delivery dates may shift.

How far the result reaches

The preview reports cross-validation on one corporate production environment.

1
Tested on one pool

Runs used shared NVIDIA H200 GPUs and varied configurations.

2
Generalization remains open

Accuracy on other clusters, software stacks and workload mixes is not established.

3
Operational impact remains unmeasured

The preview reports prediction error, not savings in GPU time or scheduling delays.

What to watchThe full paper is forthcoming; validation on independent runs and a live scheduler trial would show whether the result carries over.
The takeaway
Configuration-aware prediction beat the tested FLOP estimates on these jobs, but teams need broader validation before relying on the same accuracy for their own clusters.

Better Estimates Could Aid GPU Scheduling

Runtime predictions help cluster operators decide when to dispatch work and how to allocate limited accelerators. If a job finishes sooner than expected, GPUs may sit idle until another job starts; if it runs long, it can delay other work and shift delivery schedules. A more accurate estimate could help teams plan allocations around those risks, especially on shared clusters where jobs compete for resources.

The reported error gap is substantial within this dataset, but prediction accuracy is not the same as demonstrated operational savings. The preview reports model performance, not a controlled scheduler deployment. It does not quantify whether the approach reduced GPU-hours, costs, idle intervals or delivery delays. Those outcomes would need to be measured separately.

Products Worth Considering

DJI Mini 4K Drone Combo with 4K Camera, 3-Axis Gimbal, and Two Batteries

DJI Mini 4K Drone Combo with 4K Camera, 3-Axis Gimbal, and Two Batteries

As an affiliate, we earn on qualifying purchases.

Why FLOPs Miss Parts of Runtime

A common training estimate approximates dense Transformer compute as six times the parameter count and training-token count, then divides by GPU count, peak throughput and an assumed MFU. MFU represents the share of theoretical peak throughput achieved in practice; the preview describes a typical range of roughly 30% to 60%. This calculation focuses on training compute and depends on assumptions about hardware utilization.

The study’s target includes more than the training loop. Setup, model loading, tokenization, scheduled evaluations, checkpoint uploads and final storage can all add wall-clock time. Hugging Face says these costs vary with configuration and can compete for network and storage capacity in a multi-tenant environment. The researchers included scheduled evaluation tokens in the baseline and tested several MFU values rather than relying on one assumption.

The preview says the team compared several candidate approaches before selecting CatBoost and tuning its features. It describes the records as production runs, but the evaluation was cross-validation within the same dataset, not a test across independently collected clusters. The announced full paper is not yet available.

“Everything is known before dispatch.”

— Hugging Face study preview

How Well Results Travel

The preview does not establish whether the reported accuracy will hold on other GPU types, clusters, software stacks or workload mixes. Its 272 jobs came from one corporate production environment, and cross-validation on those records does not substitute for testing on a separately collected site or later time period.

The full paper may provide more information about the dataset, evaluation design and feature selection. The preview does not report a live scheduling trial or enough detail to judge how performance changes as workloads and cluster conditions shift. It is also unclear how often the predictor would need retraining. The reported scores are evidence of prediction performance on this dataset, not proof of a general-purpose runtime advantage.

Full Paper and External Testing

Hugging Face says it plans to post the full version on arXiv, but has not provided a publication date. Further details in that paper could help readers assess the dataset and validation methods. Independent testing across different periods or environments would show whether the prediction gap persists beyond the production pool used here.

A subsequent operational trial could test whether feeding estimates into a scheduler leads to fewer delays or idle GPU intervals. Until those results are available, the practical scheduling benefit remains unmeasured, and the study’s results should be read as a promising finding from a single environment rather than a settled result across GPU clusters.

Key Questions

What did Hugging Face report?

Its study preview says a CatBoost model predicted runtimes for 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation.

How did the FLOP-based estimate perform?

The best tested setting, at 30% MFU, had a 12.41-hour mean absolute error. The preview also reports errors of 13.01 hours at 45% MFU and 14.62 hours at 60% MFU.

What does the runtime measurement include?

It measures wall-clock time from when a job receives GPUs until it finishes. Queue time is excluded; setup and other job-lifecycle overheads can contribute to the measured duration.

Does the result prove the model will work on other clusters?

No. The jobs came from one corporate production environment, and the reported cross-validation does not establish accuracy on other hardware, software stacks or independently collected workloads.

Has the predictor been shown to save GPU time?

The preview reports prediction errors, not a controlled deployment’s effects. It does not quantify GPU time, cost, idle intervals or delays saved by using the model in a scheduler.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Why Claudeforce Could Be The Ultimate AI Solution For CRM Systems

Salesforce and Anthropic announced Claudeforce, integrating AI into CRM. Details on capabilities, availability, and security are still pending.

After South Korea’s AI Cheating Admission, What Should A Tax Certificate Signal?

South Korea’s top tax exam scorer admitted using a camera and AI to cheat. Authorities are investigating as exam security tightens before the Suneung.

The Impact Of ChatGPT Ads On Europe’s AI Marketing Strategies

OpenAI has announced the regional expansion of ChatGPT Ads into Europe, with details on countries, launch dates, and ad formats still pending.

The Debate Over Watermarks: Will They Restrict Claude Users At Work And School?

Anthropic introduces machine-readable watermarks for Claude AI content, raising concerns about detection in educational and workplace settings.