3

I know that gradient descent allows you to find the local minimum of a function. What I don't know is what exactly that function IS. It's usually called the loss function (and, in general, objective function) and often denoted as $\mathcal{L}$ or $L$ (or something like that, i.e. it is not really important how you denote it). The specific function used as a ...


3

Welcome to AI Stack exchange! You're right, as the network is initialised randomly, the resultant function is essentially impossible to get your head around. This is because most of the time the network has >4 dimensions (4 can be graphed with some effort and a lot of color), and as such is literally beyond human comprehension via graphing. So what do we ...


3

The method you propose is already known, its basically a numerical approximation to the gradient. It is not used to train neural networks because its well... an approximation. You still need to do two forward passes to get an approximation, which introduces noise and might make the training process fail. Using backpropagation to compute the gradient is an ...


2

There is nothing stopping you, you can setup Dense Neural Networks to have any size inputs or outputs (simple proof is to imagine a single layer NN with no activation is just a linear transform and given input dim $n$ and output dim $m$, it's just a matrix of $n$ x $m$, trivially this works though with any number of hidden layers) The better question is ...


2

It depends. It could give you a boost or it could not. Intuitively I would expect it to actually hurt performance if the network is initialized correctly (I think the optimizer is less of a bottleneck because they will have the same effect in both approaches). Ideal World: We optimize the network as a whole to gain better course grained features over the ...


1

In a simple feed-forward network, each artificial neuron has a separate bias value. This allows for greater flexibility for the output layer function than if each neuron had to use a single whole-layer bias. Although not an absolute requirement, without this arrangement it may become very hard to approximate some functions. Moving from a bias vector to a ...


1

Mathematical Exploration let $\Theta^+$ be the pseudo-inverse of $\Theta$. Recall, that if a vector $\boldsymbol v \in R(\Theta)$ (ie in the row space) then $\boldsymbol v = \Theta^+\Theta\boldsymbol v$. That is, so long as we select a vector that is in the rowspace of $\Theta$ then we can reconstruct it with full fidelity using the pseudo inverse. Thus, ...


Only top voted, non community-wiki answers of a minimum length are eligible