What is the fundamental difference between NN for classifying data and generating data?

Most examples show how neural networks can be used to classify data. Like is it an image of a dog or a cat. However, there are applications where NN are used to create images and even write short stories.


The formal name for this difference is "generative" vs "discriminative" models.

By default, a supervised learning process using a simple feed-forward neural network and a set of training data with expected answers will produce a discriminative model. It is hard to use such a model to generate content directly.

Differences between discriminative and generative models tend to be at the architecture level or deeper, although a few generative techniques are simple variants or re-purposing of discriminative networks. There is no single "fundamental" difference in terms of NN design. About the only common theme is that generative models are more complex and harder to work with than discriminative ones.


  • Probably the most popular currently for image generation are Generative Adversarial Networks or GANs. These train a Generator to create an image from random inputs, alongside a Discriminator that tries to spot fakes compared to real images. Training them together results in a generator network that gets progressively better at creating fake data.

  • Also suitable for image generation are Variational Autoencoders (VAEs) which learn essentially by compressing a set of images down into a small representation, and as a result become able to "uncompress" similar representations.

  • There are also VAEGANs, which combine VAEs and GANs

  • There are other models which generate examples from a dataset. For example, Restricted Boltzmann Machines (RBMs) are similar to neural networks, but the neurons fire randomly, and RBMs require a different training process to NNs.

  • GANs, VAEs, VAEGANs, RBMs can be used to generate data in any simple non-sequential datasets, they are not restricted to just images, but recent work with GANs for example has excelled there.

  • For a different kind of generator, you can look at how Deep Dream works. The interesting thing here is that it is a modification of a feed forward network that has been trained by supervised learning. Essentially to run Deep Dream, you take an existing image and change it so that it maximises some internal part(s) of the existing neural network, "training" the image as if it were a set of weights.

    • The difference between Deep Dream and other generative techniques is that it does not generate content that could be in the training set - it is no good for creating "realistic" images.


  • Recurrent Neural Networks, and specifically Long Short-Term Memory (LSTM) networks can be used to model word sequences by training them to predict the next word. Because the resulting model is probabilistic (e.g. chance of next work being "Hello" is ~0.001%) then a simple method to make this generative is to sample from the predictions randomly, and then feeding them back into the sequence to see what it will predict next. There is a great blog post about this technique by Andrei Karpathy.

Audio and Music

  • DeepMind's WaveNet architecture is similar to a CNN predicting the next samples of audio from a given input. It can be used in speech or music generation.

  • LSTMs are a popular choice here, as well as for language models.

One important caveat. Nearly all these models are hard to understand, work with and train, compared to simpler supervised techniques. Probably the easiest to get to grips with if you want to understand them well enough to implement your own versions, are Deep Dream, VAEs and LSTMs.

It is also worth noting that there are many other ways to generate content than neural networks. For instance, sound and music generation has a long history completely independent of recent AI developments.

| improve this answer | |
  • 1
    $\begingroup$ I would say that the sentence "The formal name for this difference is "generative" vs "discriminative" models." is a little misleading. Generative and discriminative models are not necessarily NNs (and given the context of the question). I would rephrase it to say that, in general, there two families of algorithms that can be used for classification: generative and discriminative models. Maybe you meant something else. $\endgroup$ – nbro Feb 28 '19 at 15:45
  • $\begingroup$ @nbro: Except you would not use generative models for classification. I'll try to re-phrase the first sentence. $\endgroup$ – Neil Slater Feb 28 '19 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.