Most tutorials make LLM distillation look like a weekend project. Forty lines of Hugging Face code, a BERT to DistilBERT swap, done. The real pipeline is a different scale of work. You pick a teacher you are legally allowed to use, generate a synthetic dataset that does not poison the student, train with LoRA, and prove the student did not quietly regress on the inputs your users actually send.
The training part is easy 20 percent. The data and evaluation work is the 80 percent that decides whether the project delivers or stalls. This is also why most teams who go all-in on model distillation services end up needing help on the synthetic data and evaluation layers, not the training loop. Before walking through how to distill an LLM end to end, it helps to settle what teacher-student distillation actually means in practice.
What Teacher-Student Distillation Actually Means
A large teacher model produces outputs on a set of inputs, and a smaller student is trained to reproduce them. The student inherits most of the teacher’s task performance at a fraction of the inference cost. White-box distillation uses the teacher’s full probability distribution over tokens, which carries more learning signal but requires logit access, so open-weights teachers only. Black-box distillation uses only the text the teacher returns, which works with any hosted API but gives less signal per sample.
That is what teacher student model training looks like at the conceptual level. The proof point everyone remembers is DistilBERT. The Hugging Face team showed it could shrink BERT by 40% and run 60% faster while keeping 97% of its language understanding on the GLUE benchmark. That is the ceiling. The floor, when teams skip the steps below, is a small model that is confidently wrong on the inputs that matter. Distillation is also a different call from retrieval, and our fine-tuning vs RAG decision framework is the better starting point if you have not made that choice yet.
Step 1: Pick the Teacher
The teacher decides two things you cannot change later: how good the student can become, and whether you are allowed to ship it. Both questions deserve a real answer before you generate a single token of training data.
Capability Fit
The student inherits the teacher’s ceiling, not your ambition. If the teacher hallucinates on your domain or fumbles your edge cases, the student will reproduce those failures faster and cheaper. Run your hardest 50 prompts through the candidate teacher before committing. If the teacher scores 70 percent, the student is not going to score 90. Pick a teacher that is already strong on the specific task you want to compress, not the model with the loudest benchmark numbers.
Licensing
This is the section most tutorials skip. After January 2025, the closed-model providers tightened their terms in public. OpenAI, Anthropic, Mistral, and xAI all include clauses that prohibit using their outputs to train competing models, with “competing” interpreted broadly. According to Financial Times, OpenAI found evidence of DeepSeek using its models to train a rival, and the dispute became the reference case every legal team now cites. In February 2026, Anthropic publicly accused DeepSeek, Moonshot and MiniMax of distillation attacks on Claude, making clear that providers are watching for this.
Open-weights teachers carry their own terms. The Llama community license has commercial-use thresholds, several research releases ban commercial use outright, and Apache 2.0 models are the cleanest path if your legal review is risk-averse. The short version: read the model card, read the API terms, and document the decision in writing. Every later legal review will ask which teacher you used and what its license allowed.
Step 2: Generate the Synthetic Dataset
This is where the project is won or lost. The student learns whatever is in the data, including the teacher’s biases, hallucinations, and stylistic tics. A typical LLM distillation example in research papers uses tens of thousands of carefully filtered samples, and the time spent on filtering usually exceeds the time spent on training.
Seed Inputs and Volume
Start with real domain prompts: production logs, support tickets, redacted user queries, internal documents. Diversity matters more than raw volume past a certain point. For a narrow task, 1,000 to 5,000 well-curated seeds often beat 50,000 noisy ones. For a broader instruction-following student, plan for 20,000 to 100,000. If real prompts are not available, generate seeds with a separate model and have a human review the distribution before scaling up.
Three Generation Strategies
Each method trades effort against signal quality.
Soft-label distillation
Logits (open weights only)
High
Classification, narrow tasks
Output-only distillation
Any hosted API
Low
Generative tasks, general use
Distilling Step-by-Step
Any teacher that can explain
Medium
Reasoning tasks, low-data regimes
Distilling Step-by-Step is worth knowing. The teacher generates a rationale alongside the answer, and the student trains on both. Google and Snorkel researchers showed this can match a larger model with significantly less data, which matters when your seed pool is small.
Filtering and Expert Review
The unglamorous part of the pipeline. Dedupe and run ROUGE-L filtering for near-duplicates. Drop length and format outliers that will confuse the student. Spot-check a random 5 percent slice for hallucinations. For regulated outputs, get a domain expert to sign off on a representative sample. Expect to reject 20 to 40 percent of generations along the way. Teams that skip filtering ship students that overfit to the teacher’s quirks. For legal, medical, or financial tasks, expert review is not optional, and if that review capacity is the constraint, our AI agent development services often include this layer.
Step 3: Train the Student
The easy 20 percent. LoRA injects small low-rank adapters into existing weights and freezes the rest, dropping trainable parameters to about 1 percent of the total. QLoRA adds 4-bit quantization of the base model on top of LoRA, dropping memory needs by roughly half again. The practical outcome: a 7B student becomes fine-tunable on a single 24 GB GPU instead of a multi-GPU cluster.
Configuration that holds across most tasks:
- Rank between 8 and 64. Alpha at roughly twice the rank.
- Learning rate near 2e-4 with a cosine schedule.
- Gradient accumulation to hit your effective batch size.
- For soft-label distillation: weighted combination of KL divergence (teacher vs student distributions) and cross-entropy on hard labels, with temperature scaling on the softmax, T = 2 to 4.
One piece of advice production teams agree on: start with the smallest viable student. A 1B model takes 20 minutes per run and surfaces infrastructure problems immediately. A 70B model takes a day and hides them. Get the loop working end to end on a small model, then scale once you trust it. Teams building larger systems on top of the distilled model usually need large language model development work that goes beyond the training loop itself.
Step 4: Evaluate Properly
This is where most distillation projects quietly fail. Training completes, MMLU looks fine, the team ships, and three weeks later support tickets reveal the student is dropping on long inputs, ambiguous instructions, and anything outside the synthetic data distribution.
Public benchmarks are the weakest signal in the picture. MMLU, GLUE, and HellaSwag are useful mainly for spotting catastrophic regressions, since data contamination is endemic and the test set may have leaked into pretraining or into the teacher’s outputs. The asset that actually matters is a held-out evaluation set built from real traffic. Five hundred to two thousand cases, hand-labeled where possible, covering input length, complexity, edge cases, adversarial prompts, and the long tail your users actually send.
Use multiple methods together. Task-specific metrics like exact match or BLEU for narrow outputs. LLM-as-judge using a different model family than your teacher, to avoid same-family bias. Human spot-checks on the bottom 10 percent of judged scores, which is where regressions hide. Track per-category performance, never just the average. A student that scores 96 percent overall can score 40 percent on long inputs while the average looks healthy.
Building and maintaining this harness is more sustained engineering work than the training itself. Teams without dedicated ML infrastructure underbuild it, ship, and find the problems in production. This is where companies often bring in outside help, both for the initial build and for the regression suite that survives every model swap. Production AI development rests on this layer more than on any single model.
When Distillation Is Worth It and When It Isn't
The decision is rarely about whether distillation works. It is about whether your task and team are set up for it.
Good fit when:
- High-volume narrow task.
- Latency-sensitive product surface.
- Predictable inference cost target.
- Real production logs available for synthetic data.
A teacher student model approach pays for itself fastest when inference volume is large enough that the cost delta over a hosted API recoups the engineering investment within a quarter or two.
Poor fit when:
- The teacher does not actually perform well on your task.
- No real eval data exists.
- No domain expertise available for review.
In those cases, retrieval-augmented generation, prompt caching, or a smaller hosted model often delivers 60 to 80 percent of the cost benefit with none of the training complexity. The question of how to fine-tune small model versus retrieve versus distill deserves a structured comparison before you commit engineering time.
The Realistic Timeline
The Hugging Face quickstarts do not show this part. A production-grade pipeline for a moderately complex task takes 2 to 4 weeks of synthetic data generation and filtering, 1 week of training and iteration, and 2 to 3 weeks of evaluation harness construction, plus ongoing maintenance as the source domain shifts.
Teams that estimate purely on training compute consistently underestimate the project by a factor of three to five. The eval harness is the asset that outlives every individual model. You reuse it every time you swap the student, retrain on fresh data, or test a new teacher. Treat it as a durable investment, not the model itself.
The Launch Bar
A distilled model is ready when it passes your held-out eval suite, including the edge-case batteries, and survives two weeks of shadow traffic without surprising the team. Not when training loss looks clean. Not when MMLU is acceptable. The smallest student that clears those gates is the one worth putting into production. Resisting the urge to compress further until it has run in front of real users is usually the right call.
A distilled model is one project. Keeping the synthetic data pipeline honest, the eval suite current, and the student tuned as your domain shifts is another. Most teams handle the first comfortably and underestimate the second. If you would rather not staff both, contact us, and we can talk through which parts are worth bringing in help for.
See how we transformed a legacy AI platform into a production-grade digital growth solution.
