Generative vs Discriminative: Two Ways a Model Uses Data
The distinction that defines the field: models that learn decision boundaries versus models that learn the shape of the data itself — and why modern LLMs blur the line.
- Define generative and discriminative models in terms of the probability each one estimates
- Classify six real AI products as generative or discriminative and justify each from its training objective
- Explain why a generative model can still be used for classification, and give the reverse counter-example
- Choose generative vs discriminative for three given business problems and defend the choice
Two apps sit on the same phone. One tags your photos: "beach, dog, dinner" — it never invents a pixel, it only labels. The other, in the same minute, invents a photograph of a dog on a beach that never existed. Both are "AI", both were trained on images, but they were trained for fundamentally different mathematical jobs. That split — discriminative vs generative — is the fault line of the entire field, and every model you will meet in this course lives on one side of it.
Two different questions about data. A discriminative model learns P(label | input): given an image, how should I score each class? Its job is a boundary between classes. Logistic regression, SVMs, BERT-style classifiers, and the "is this spam?" layer in your email client are discriminative. A generative model learns P(input) (or P(input, label)): what does the data itself look like? How likely is this particular image, this sentence, this row? Because it models the whole distribution, it can sample — produce new data that never existed — and also answer questions like "how strange is this point?" (low probability = anomaly). Naive Bayes, Gaussian mixtures, GANs, VAEs, diffusion models, and every LLM are generative.
What each one buys you. Discriminative models go straight at the decision you care about; with labelled data they usually win on accuracy per unit of effort, because they spend all their capacity on the boundary. They cannot, by themselves, write a poem. Generative models spend capacity modelling everything about the data — including variation you may not care about — which is the price of being able to create, impute missing values, and detect anomalies. Generative models can also classify via Bayes' rule: if you know P(x | class) and P(class), you can compute P(class | x). The classic example is naive Bayes email filtering: count word frequencies per class, multiply, normalise — a generative model doing a discriminative job.
The modern blur, and how to classify honestly. Ask not "how is it used?" but "what objective was it trained on?" An LLM trained on next-token prediction over raw text is generative — even when you prompt it with "reply only SPAM or HAM", it generates the label as a token sequence rather than scoring classes internally. Reverse case: a semantic-search system that embeds documents and thresholds cosine similarity is doing discriminative-style separation with no generation involved. And a image classifier can be bolted onto a generative pipeline (detect, then regenerate) without changing what either model is. So the practical rule: classify by the training objective, not by the user-facing trick.
Why the generative half exploded. Discriminative deep learning peaked quietly in the 2010s (ImageNet, ad ranking, spam). The 2020s wave came from the generative side getting good enough at scale — transformers scaling next-token prediction to fluent text, diffusion scaling image synthesis to photorealism — so that sampling from P(data) now produces artifacts people actually want: essays, code, faces, music, voice. That is what "Generative AI" names: not a new mathematics of probability, but the point where modelling P(data) became useful enough to build products on. Every remaining lesson in this course is an elaboration of one of those two sentences: what the model estimates, and what it can do because of it.
| Aspect | Discriminative | Generative |
|---|---|---|
| Learns | P(label | x) — the decision boundary | P(x) or P(x, label) — the shape of the data |
| Can sample new data? | No | Yes — that is the point |
| Classic examples | Logistic regression, SVM, BERT classifier | Naive Bayes, GAN, VAE, diffusion, LLM |
| Typical modern product | Photo tagging, spam filter, fraud score | ChatGPT, Midjourney, Suno, Copilot |
| Typical strength | Accuracy on a fixed label set | Creation, anomaly scoring, flexibility across tasks |
| Typical weakness | Cannot create; needs labels | Spends capacity on detail you may not need; can hallucinate |
When someone says "we built an AI that does X", ask: was it trained to score labels, or trained to reproduce data? The answer predicts its failure modes before you see a single demo.
A generative model optimises likelihood, not correctness. Fluency is a property of P(text); truth is a property of the world. Confusing the two is the root of hallucination (covered in depth in Part II).
- 1
Step 1 — classify by objective
(a) discriminative: trained on labelled fraud/not-fraud rows to score P(fraud | features). (b) generative: trained on next-token likelihood over text, sampled to produce new text. (c) discriminative: P(topic | headline). (d) borderline but treat as discriminative separation — it scores similarity against a threshold, never samples a new ticket.
- 2
Step 2 — set up the toy naive-Bayes model
Two classes, refund requests (R) and delivery questions (D). Training counts: P(R)=0.5, P(D)=0.5. Word likelihoods: P("refund"|R)=0.40, P("money"|R)=0.30, P("refund"|D)=0.05, P("money"|D)=0.05.
P(R)=0.5 P(D)=0.5 P(refund|R)=0.40 P(money|R)=0.30 P(refund|D)=0.05 P(money|D)=0.05 - 3
Step 3 — multiply likelihoods (naive independence)
Ticket text = "refund money". Score(R) = 0.5 × 0.40 × 0.30 = 0.060. Score(D) = 0.5 × 0.05 × 0.05 = 0.00125.
Score(R) = 0.5 * 0.40 * 0.30 = 0.060 Score(D) = 0.5 * 0.05 * 0.05 = 0.00125 - 4
Step 4 — normalise to a posterior
Total = 0.06125. P(R | text) = 0.060 / 0.06125 ≈ 0.9796, P(D | text) ≈ 0.0204. A *generative* model (it modelled P(words | class)) produced a confident discriminative answer — same output shape, different internal job.
P(R|text) = 0.060 / 0.06125 = 0.98 P(D|text) = 0.02
Solution
Four labels: discriminative, generative, discriminative, discriminative-by-similarity. The naive-Bayes walk-through scores 0.98 for the refund class — proving generation and classification are jobs, not mutually exclusive model species. Classify models by objective; predict behaviour from that.
List six AI features you used this week (photo tags, autocomplete, chat, transcription, recommendation feed, fraud alert). For each, write one sentence naming the objective you believe it was trained on, and one prediction about how it fails. Then find the one case where you are genuinely unsure — and say what evidence (docs, model card, demo behaviour) would settle it.
Hint: If the system only ever outputs a decision about your input, it is probably discriminative. If it can surprise you with something new, it is generative.
Calling any tool that "uses AI" generative because it is new
Most production AI remains discriminative (ranking, scoring, tagging). "Generative" names a training objective — P(data) — not novelty or neural-network depth.
Believing an LLM doing classification is a discriminative model
It generates the label as text from a next-token distribution. The objective never changed; you rephrased the task into its generative format.
Assuming generative models are automatically better at decisions
For a fixed label set with good labels, a discriminative model is usually more accurate, cheaper and more debuggable. Generality is bought with capacity you may not need.
Equating likelihood with correctness
A generative model reports how *text-like* something is, not whether it is true. High P(x) means familiar, not verified — the seed of every hallucination story.
Differentiate generative from discriminative models. Give one example of each in a modern product, and explain why an LLM prompt "classify this as SPAM or HAM" does not make the LLM a discriminative model.
A discriminative model estimates P(label | x): it learns the boundary between classes and is typically tuned for decision accuracy (e.g. an email spam scorer trained on labelled mail). A generative model estimates P(x) or P(x, label): it learns how the data itself is distributed, which lets it sample new instances and also, via Bayes, classify (e.g. ChatGPT generating text; or naive Bayes classifying from word likelihoods). Prompting an LLM to output SPAM/HAM changes the *task presentation*, not the training objective: the model still scores the next token from its learned text distribution and samples a label string. It never learned class-conditional boundaries; it learned text continuation and is being used as a classifier through prompting — which works, but inherits generative failure modes (fluency without calibration, sensitivity to phrasing).
Source: Standard ML interview screen; also a common MLOps/GenAI product-design discussion question.
- Discriminative models learn P(label | x) — a boundary; generative models learn P(x) — the shape of the data, and can sample.
- Generative models can classify (Bayes), but discriminative models cannot generate — that asymmetry is the whole difference.
- Classify by training objective, not by how the product is used: prompting an LLM to label text does not make it discriminative.
- The Generative AI wave = P(data) modelling scaled (transformers + diffusion) until sampling produced useful artifacts.
- Likelihood is not truth: generative models measure text-likeness, which is why hallucination is possible at all.