# How can neural networks be used to generate rather than classify?

In my experience with Neural Nets, I have only used them to take input vectors and return binary output.

But, here in a video, https://youtu.be/ajGgd9Ld-Wc?t=214, Kai Fu Lee, renowned AI Expert shows a deep net which takes thousands of samples of Trump's speeches and generates output in the Chinese Language.

In short, how can deep nets/neural nets be used to generate output rather than giving answer yes or no? Additionally, how are these nets being trained? Can anyone here provide me a simple design to nets that are capable of doing that?

Think of a neural network as a universal function approximator (With infinite width under a set of constraints this is actually provable). Now when discussing generation in the context you have provided, you essentially want to draw from some distribution $$p(y|c)$$ where $$y$$ is your output and $$c$$ is your context or input.

Theorem: For any distribution $$\Omega$$, if we take $$z \sim \mathcal{N}(0,I)$$, there exists a function $$f$$ where $$f(z) \sim \Omega$$.

Given the above theorem (for the purposes of this post I don't need to prove it, but its very similar to the universal approximation theorem proof) and if we take neural networks as a pseudo-universal function approximator, if we have a valid objective or training procedure that can learn the parameters of $$f$$, sampling is as easy as sampling $$\mathcal{N}(0,I)$$ and then applying $$f$$.

So the trick really is finding a good training procedure, and this is where you see GANs, VAEs and other models/schemes come into play.

Everything I've said above works really well when there isn't autocorrelation like in text, but when there is, the above methodology would result in a combinatorially large output space which isn't realistic with a vocabulary size usually spanning somewhere between a couple thousand and a couple hundred thousand. So to handle this they model the joint by taking advantage of that autocorrelation by modeling the joint probability as its bayesian decomposition. $$p(\vec w) = p(w_0)\prod_{i=1}^{N-1}p(w_i|w_{
Now that there is a framework to efficiently model this type of output, were back into the position as before where we are looking for clever training schemes. In this case you'll see commonly RNN's or other sequential model training with teacher forcing (@nbro described this in his answer too), or using GAN like compositions using either reinforcement learning to handle the lack of differentiability in sampling or using approximations like Gumbel-Softmax or Intermediate Loss Sampling (method I actually developed)

• I think I really need to pass high school doing mathematics before reaching the discussion to the level of mathematics you are talking... – user27450 Oct 18 '19 at 14:18
• Look into basic probability and I think your golden for the understanding. If you want something to get your hands dirty, just google "GAN GitHub" and you'll find tons of repos for starting point of generative models. Add more specifics to your search if your looking for something more specific. Being in high school shouldn't stop you-- Its actually great you're getting into this so early! – mshlis Oct 18 '19 at 14:38
• hehe... I think I'm too much addicted to doing AI using C++ , Python... I've recently made an AI that generates Music using Long Term/Short Term Memory manipulation using AI, wanted to add some lyrics but I needed a better voice, so, I'm now learning to generate voice using AI. – user27450 Oct 18 '19 at 14:59
• What is the meaning of $f(z)$~ $\omega$? – user27450 Oct 18 '19 at 15:00
• @AbhasKumarSinha: It means that if you have some imagined target distribution over your data $\Omega$ that you want to generate but don't have a random function for it, then you can create that random function by mapping from the normal distribution in $z$. The function $f()$, if well trained, and fed with samples from the normal distribution as inputs, will create samples over your output space - which is what you want, a random generator for your target data. – Neil Slater Oct 18 '19 at 17:36

If the output can either be yes or no, then you have a discrete and binary output, so this problem is called binary classification, that is, it is the task of classifying (or categorizing) the input into one of two categories (or classes). You can also have a neural network with an output that can take more than two possible discrete values, which is can be used to solve a multi-class classification problem. For example, a neural network that outputs a sentence, which is composed of $$n$$ words, where $$n>1$$. In general, the output does not necessarily need to take a value from a set of discrete values (or classes), but it can also take a numeric value (e.g. a floating-point number). In that case, the problem is called regression. For example, the task of predicting the height of a person (a numeric value) given a picture of the same.

There are different types of neural networks. The simplest neural network is either a perceptron (if you consider it a neural network) or a multi-layer feed-forward neural network, that is, a neural network with only forward connections, with possible multiple layers. There are also convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which are more sophisticated neural networks that are more suited for processing respectively imagery or sequences. There are also generative neural networks (e.g. variational auto-encoders), which are trained to learn a distribution, from which you can then sample.

In your specific example, the sentence could have been generated with a recurrent neural network or a generative model (or a combination of both). More precisely, a recurrent generative model could have been trained to learn the rules of either the English or Chinese language. Then you sample from this distribution to generate sentences. In principle, a sentence could also be generated with simpler neural networks (such as a multi-layer perceptron), but, in practice, this may be more inefficient.

• The term generative model is reserved for a statistical model which is equal to a normal neural network. It is – in opposition to the name – not able to produce audio information, but it's a tool for regression analysis. – Manuel Rodriguez Oct 17 '19 at 17:55
• @ManuelRodriguez What is a normal neural network? There isn't such a thing. It is true that VAEs can be implemented with MLPs, but, in this case, I mentioned VAEs (which are generative models) because they are relevant to the question, which is about the generation of sentences. – nbro Oct 17 '19 at 21:59

A neural network can do much more than only output the audio information. It's even possible to convert text to a visual talking head in which the animated character can speak and gets animated at the same time. This extra features allows to explain, what the underlying principle is. Befure a richful audio/video output can be generated, a parametric model is needed, which takes as input a value and produces output data. This is realized with domain specific knowledge which is encoded in a software.

The model for generating the output shouldn't be mistaken with a neural network model. A talking head model is a formalized representation of audio and video information. It's an animated puppetry which can be controlled by joystick movements. Creating such models can be done with object oriented programming and it's equal to program a graphic/sound-engine. It gets as input a textual command and produces as output a H.264 video file.