What is the difference between a generative and a discriminative algorithm?

Question

What is the difference between a generative and a discriminative algorithm?

Answer 1

The short answer

Many of the answers here rely on the widely-used mathematical definition [1]:

• Discriminative models directly learn the conditional predictive distribution p(y|x).
• Generative models learn the joint distribution p(x,y) (or rather, p(x|y) and p(y)).
  • Predictive distribution p(y|x) can be obtained with Bayes' rule.

Although very useful, this narrow definition assumes the supervised setting, and is less handy when examining unsupervised or semi-supervised methods. It also doesn't apply to many contemporary approaches for deep generative modeling. For example, now we have implicit generative models, e.g. Generative Adversarial Networks (GANs), which are sampling-based and don't even explicitly model the probability density p(x) (instead learning a divergence measure via the discriminator network). But we call them "generative models” since they are used to generate (high-dimensional [10]) samples.

A broader and more fundamental definition [2] seems equally fitting for this general question:

• Discriminative models learn the boundary between classes.
  • So they can discriminate between different kinds of data instances.
• Generative models learn the distribution of data.
  • So they can generate new data instances.

Image source

---

A closer look

Even so, this question implies somewhat of a false dichotomy [3]. The generative-discriminative "dichotomy" is in fact a spectrum which you can even smoothly interpolate between [4].

As a consequence, this distinction gets arbitrary and confusing, especially when many popular models do not neatly fall into one or the other [5,6], or are in fact hybrid models (combinations of classically "discriminative" and "generative" models).

Nevertheless it's still a highly useful and common distinction to make. We can list some clear-cut examples of generative and discriminative models, both canonical and recent:

• Generative: Naive Bayes, latent Dirichlet allocation (LDA), Generative Adversarial Networks (GAN), Variational Autoencoders (VAE), normalizing flows.
• Discriminative: Support vector machine (SVM), logistic regression, most deep neural networks.

There is also a lot of interesting work deeply examining the generative-discriminative divide [7] and spectrum [4,8], and even transforming discriminative models into generative models [9].

In the end, definitions are constantly evolving, especially in this rapidly growing field :) It's best to take them with a pinch of salt, and maybe even redefine them for yourself and others.

---

Sources

1. Possibly originating from "Machine Learning - Discriminative and Generative" (Tony Jebara, 2004).
2. Crash Course in Machine Learning by Google
3. The Generative-Discriminative Fallacy
4. "Principled Hybrids of Generative and Discriminative Models" (Lasserre et al., 2006)
5. @shimao's question
6. Binu Jasim's answer
7. Comparing logistic regression and naive Bayes:
   • cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf
   • "On Discriminative vs. Generative classifiers"
   • Comment on "On Discriminative vs. Generative classifiers"
8. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/04/DengJaitly2015-ch1-2.pdf
9. "Your classifier is secretly an energy-based model" (Grathwohl et al., 2019)
10. Stanford CS236 notes: Technically, a probabilistic discriminative model is also a generative model of the labels conditioned on the data. However, the term generative models is typically reserved for high dimensional data.

Answer 2

An addition informative point that goes well with the answer by StompChicken above.

The fundamental difference between discriminative models and generative models is:

Discriminative models learn the (hard or soft) boundary between classes

Generative models model the distribution of individual classes

Edit:

A Generative model is the one that can generate data. It models both the features and the class (i.e. the complete data).

If we model P(x,y): I can use this probability distribution to generate data points - and hence all algorithms modeling P(x,y) are generative.

Eg. of generative models

• Naive Bayes models P(c) and P(d|c) - where c is the class and d is the feature vector.

  Also, P(c,d) = P(c) * P(d|c)

  Hence, Naive Bayes in some form models, P(c,d)

• Bayes Net

• Markov Nets

A discriminative model is the one that can only be used to discriminate/classify the data points. You only require to model P(y|x) in such cases, (i.e. probability of class given the feature vector).

Eg. of discriminative models:

• logistic regression

• Neural Networks

• Conditional random fields

In general, generative models need to model much more than the discriminative models and hence are sometimes not as effective. As a matter of fact, most (not sure if all) unsupervised learning algorithms like clustering etc can be called generative, since they model P(d) (and there are no classes:P)

PS: Part of the answer is taken from source

Answer 3

This article helped me a lot in understanding the concept.

In summary,

• Both are probabilistic models, meaning they both use probability (conditional probability , to be precise) to calculate classes for the unknown data.
• The Generative Classifiers apply Joint PDF & Bayes Theorem on the data set and calculate conditional probability using values from those.
• The Discriminative Classifiers directly find Conditional probablity on the data set

Some good reading material: conditional probability , Joint PDF

Answer 4

A generative algorithm models how the data was generated in order to categorize a signal. It asks the question: based on my generation assumptions, which category is most likely to generate this signal?

A discriminative algorithm does not care about how the data was generated, it simply categorizes a given signal.

Answer 5

All previous answers are great, and I'd like to plug in one more point.

From generative algorithm models, we can derive any distribution; while we can only obtain the conditional distribution P(Y|X) from the discriminative algorithm models(or we can say they are only useful for discriminating Y’s label), and that's why it is called discriminative model. The discriminative model doesn't assume that the X's are independent given the Y($X_i \perp X_{-i} | Y$) and hence is usually more powerful for calculating that conditional distribution.

Answer 6

A generative algorithm model will learn completely from the training data and will predict the response.

A discriminative algorithm job is just to classify or differentiate between the 2 outcomes.