3
$\begingroup$

At least at some level, maybe not end-to-end always, but Deep Learning always learns a function, essentially a mapping from a Domain to a Range. The Domain and Range, at least in most cases, would be multi-variate. So when a model learns a mapping, considering every point in the domain-space has a mapping, does it try to learn a continuous distribution based on the training-set and its corresponding mappings, and map unseen examples from this learned distribution? Could this be said about all predictive algorithms?

If yes, then could binary classification be compared to having a hyper-plane (as in Support Vector Classification) in a particular kernel-space, and could the idea of classification problems using hyper-planes be extended in general to any Deep Learning problem learning a mapping?

Would also explain why Deep Learning needs a lot of data and why it works better mostly then other Learning algorithms for simple problems.

Any answer, even personal opinions on this would be appreciated!

$\endgroup$
3
$\begingroup$

Well, there are some questions here...

Does it (Deep Learning) try to learn a continuous distribution based on the training-set and its corresponding mappings, and map unseen examples from this learned distribution?

Yes. Talking about Deep Artificial Neural Networks, they use continuous activation functions in each neuron, therefore, the output is also a continuous function. The issue with the unseen examples is that they need to have similar examples in the training set, otherwise the weights and bias of the network will not be tuned in the regions of space around the unseen example. Imagine a Neural Network learning a function y = x, if we only present values between 0 and 10 during training, we should expect it to only make good predictions for y for values of x ranging from 0 to 10. It doesn't mean that it won't predict for other values, but the predictions will not be so accurate or nowhere close to the expectations. That is because the network is not trying to guess what was the function used to generate y, but it is simply trying to adjust its parameters to make its internal functions generate the expected y for the given x. That is why Deep Neural Networks require a lot of data. In a unidimensional space is easier to provide examples that cover the subset of the domain we want our network to learn. When we use multidimensional space, we need a lot more examples to have a good representation of the hyperspace used as domain.

Could this (map unseen examples) be said about all predictive algorithms?

Yes, it should. Otherwise, the algorithm would not able to generalize well. A good predictive algorithm is the one that can predict unseen examples using fewer training samples.

Could the idea of classification problems using hyper-planes be extended in general to any Deep Learning problem learning a mapping?

In the case of Deep Neural Networks, the result is more like the probability of a value given to the input belonging to a class. For binary classification, the network will have a single output. This output is usually modulated by a sigmoid function, that ranges between 0 and 1. The output can be interpreted as the probability of belonging to one specific class of the two possible classes. To know the probability of the other class we this subtract it from 1. For three or more classes, we will have three or more outputs ranging from 0 to 1, and each output is the probability of belonging to one of the classes. In this case, the outputs are also normalized by a softmax function, that basically guarantees that the sum of all output is equal to 1, as a probability distribution.

Would also explain why Deep Learning needs a lot of data and why it works better mostly then other Learning algorithms for simple problems.

Already partially explained... The need for a lot of data is to have a good representation of the hyperplane used as the domain. The Deep Neural Networks work well because their power to represent different models. They are a very 'flexible' functions that can be bent to approximate the relation existent between the data in the training set and the expected target. Simpler algorithms, as linear models, for instance, have less representation power, they are limited to a smaller set of models. Even though many models can be linearly approximated (because the input and output almost follow a linear relation), the neural network will be able to learn the nuances of the dataset better. This can also be the curse of neural networks, because they may try to learn every detail of the training set that wasn't really relevant and True for other cases, and that's is overtraining... but that is for another time.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.