# How can neural networks be used to generate rather than classify?

In my experience with Neural Nets, I have only used them to take input vectors and return binary output.

But, here in a video, https://youtu.be/ajGgd9Ld-Wc?t=214, Kai Fu Lee, renowned AI Expert shows a deep net which takes thousands of samples of Trump's speeches and generates output in the Chinese Language.

In short, how can deep nets/neural nets be used to generate output rather than giving answer yes or no? Additionally, how are these nets being trained? Can anyone here provide me a simple design to nets that are capable of doing that?

Think of a neural network as a universal function approximator (With infinite width under a set of constraints this is actually provable). Now when discussing generation in the context you have provided, you essentially want to draw from some distribution $$p(y|c)$$ where $$y$$ is your output and $$c$$ is your context or input.

Theorem: For any distribution $$\Omega$$, if we take $$z \sim \mathcal{N}(0,I)$$, there exists a function $$f$$ where $$f(z) \sim \Omega$$.

Given the above theorem (for the purposes of this post I don't need to prove it, but its very similar to the universal approximation theorem proof) and if we take neural networks as a pseudo-universal function approximator, if we have a valid objective or training procedure that can learn the parameters of $$f$$, sampling is as easy as sampling $$\mathcal{N}(0,I)$$ and then applying $$f$$.

So the trick really is finding a good training procedure, and this is where you see GANs, VAEs and other models/schemes come into play.

Everything I've said above works really well when there isn't autocorrelation like in text, but when there is, the above methodology would result in a combinatorially large output space which isn't realistic with a vocabulary size usually spanning somewhere between a couple thousand and a couple hundred thousand. So to handle this they model the joint by taking advantage of that autocorrelation by modeling the joint probability as its bayesian decomposition. $$p(\vec w) = p(w_0)\prod_{i=1}^{N-1}p(w_i|w_{
Now that there is a framework to efficiently model this type of output, were back into the position as before where we are looking for clever training schemes. In this case you'll see commonly RNN's or other sequential model training with teacher forcing (@nbro described this in his answer too), or using GAN like compositions using either reinforcement learning to handle the lack of differentiability in sampling or using approximations like Gumbel-Softmax or Intermediate Loss Sampling (method I actually developed)

• I think I really need to pass high school doing mathematics before reaching the discussion to the level of mathematics you are talking...
– user27450
Oct 18 '19 at 14:18
• Look into basic probability and I think your golden for the understanding. If you want something to get your hands dirty, just google "GAN GitHub" and you'll find tons of repos for starting point of generative models. Add more specifics to your search if your looking for something more specific. Being in high school shouldn't stop you-- Its actually great you're getting into this so early! Oct 18 '19 at 14:38
• hehe... I think I'm too much addicted to doing AI using C++ , Python... I've recently made an AI that generates Music using Long Term/Short Term Memory manipulation using AI, wanted to add some lyrics but I needed a better voice, so, I'm now learning to generate voice using AI.
– user27450
Oct 18 '19 at 14:59
• What is the meaning of $f(z)$~ $\omega$?
– user27450
Oct 18 '19 at 15:00
• @AbhasKumarSinha: It means that if you have some imagined target distribution over your data $\Omega$ that you want to generate but don't have a random function for it, then you can create that random function by mapping from the normal distribution in $z$. The function $f()$, if well trained, and fed with samples from the normal distribution as inputs, will create samples over your output space - which is what you want, a random generator for your target data. Oct 18 '19 at 17:36

If the output can either be yes or no, then you have a discrete and binary output, so this problem is called binary classification, that is, it is the task of classifying (or categorizing) the input into one of two categories (or classes). You can also have a neural network with an output that can take more than two possible discrete values, which is can be used to solve a multi-class classification problem. For example, a neural network that outputs a sentence, which is composed of $$n$$ words, where $$n>1$$. In general, the output does not necessarily need to take a value from a set of discrete values (or classes), but it can also take a numeric value (e.g. a floating-point number). In that case, the problem is called regression. For example, the task of predicting the height of a person (a numeric value) given a picture of the same.

There are different types of neural networks. The simplest neural network is either a perceptron (if you consider it a neural network) or a multi-layer feed-forward neural network, that is, a neural network with only forward connections, with possible multiple layers. There are also convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which are more sophisticated neural networks that are more suited for processing respectively imagery or sequences. There are also generative neural networks (e.g. variational auto-encoders), which are trained to learn a distribution, from which you can then sample.

In your specific example, the sentence could have been generated with a recurrent neural network or a generative model (or a combination of both). More precisely, a recurrent generative model could have been trained to learn the rules of either the English or Chinese language. Then you sample from this distribution to generate sentences. In principle, a sentence could also be generated with simpler neural networks (such as a multi-layer perceptron), but, in practice, this may be more inefficient.