I am trying to understant how it works. How do you teach it say, to add 1 to each number it gets. I am pretty new to the subject and I learned how it works when you teach it to identify a picture of a number. I can understand how it identifies a number but I cant get it how would it study to perform addition? I can understand that it can identify a number or picture using the pixels and assigning weights and then learning to measure whether a picture of a number resembling the weight is assigned to each pixel. But i can't logically understand how would it learn the concept of adding a number by one. Suppose I showed it thousands of examples of 7 turning to 8 152 turning into 153 would it get it that every number in the world has to be added by one? How would it get it having no such operation of + ? Since addition does not exist to its proposal then how can it realize that it has to add one in every number? Even by seeing thousands of examples but having no such operation of plus I cant understand it. I could understand identifying pixels and such but such an operation I cant get the theoretical logic behind it. Can you explain the logic in layman terms?
-
2$\begingroup$ "having no such operation of plus..." why do you say this? Neural networks perform linear combinations of inputs, followed by non-linearities. Of course they have the + operation! $\endgroup$– firionAug 27, 2018 at 11:51
-
1$\begingroup$ If you have a problem in understanding my answer do not worry, I was also a beginner and did not understand ReLu and stuff, you can always clarify $\endgroup$– user9947Aug 27, 2018 at 13:48
2 Answers
Welcome to AI.SE @bilanush.
Here's an example approach that might make things clearer. There are other ways to train a neural network to do this however.
In your earlier example with an image, you probably noticed that the network receives the image as a series of values, representing each pixel in the image. The network then learns which of a series of output neurons should be active in response to a given set of pixel values. Those output neurons, when read in an appropriate way, correspond to the correct label for the image. The difference between the set of outputs that should have been active, and the set that were active, forms the basis of the error signal that allows the network to learn.
You've probably heard that computers represent numbers with binary digits. So you could think of the number 16 as being: 00010000 in "8-bit binary". In 16-bit binary, this number would be 0000000000010000, and so on.
So one way of viewing your problem is a function mapping binary inputs to binary outputs (very similar to labelling a black-and-white image). For instance, the input 00010000 (16) should produce the output 00010001 (17). The input 00100011 (35) should produce the output 00100100 (36), and so on.
As before, you will have a set of output neurons. In this case, it should be wide as the set of input neurons. As before, the error signal is the difference between the expected inputs and outputs.
As to the question of how they can learn this function "without plus", in fact the individual neurons in a network perform just two operations: addition of their inputs, and a non-linear transformation of the sum. It has been proven that these are sufficient to learn any function from inputs to outputs, as long as the network contains 3 layers or more, and the as long as the middle layer is wide enough, but here it should be easy to see how addition might emerge.
-
$\begingroup$ Although it is a nice viewpoint, it will not work for unseen examples $\endgroup$– user9947Aug 27, 2018 at 13:36
-
$\begingroup$ @DuttaA I think you are mistaken: a sufficiently wide and deep network can learn this pattern and extrapolate to unseen examples. In particular, since the pattern is a series of full-adder gates, it can be done simply with a network that has depth and width proportionate to the number of bits. I agree with your answer that there are better approaches to solving this problem, but they may be harder for a novice to follow. $\endgroup$ Aug 28, 2018 at 1:45
This is what we call a regression problem. Although @John has provided a novel method I do not think it will work since you are decomposing the number into its minimal representation, so teaching it will be quite tough due to long term dependencies like 0111111 will change to 100000, so you have to train on almost all examples, with 0 actual learning.
Let's see your problem from a different viewpoint. Why are you thinking only of integers? Your problem can be generalised to approximating this curve:
This is clearly an example of $W.T * X + B$. Single input node will feed into 2 output nodes of Leaky ReLu activation, and a bias node. The bias node will be your $c$ and the 2-3 Leaky ReLu's will adjust their weights to create a straight line. Training this might be a problem (with co-adaptation between nodes), but mathematically a solution will be achieved by this Neural Net structure.
Also it is better to train on real values also, for better and finer weight adjustment (although theoretically for a single independent variable $x$ you should just need 3-4 values to make this Neural Net learn, but who knows?)
NOTE: The approximation in the negative region might not be that great.
