3
I know that gradient descent allows you to find the local minimum of a function. What I don't know is what exactly that function IS.
It's usually called the loss function (and, in general, objective function) and often denoted as $\mathcal{L}$ or $L$ (or something like that, i.e. it is not really important how you denote it). The specific function used as a ...
3
Welcome to AI Stack exchange!
You're right, as the network is initialised randomly, the resultant function is essentially impossible to get your head around. This is because most of the time the network has >4 dimensions (4 can be graphed with some effort and a lot of color), and as such is literally beyond human comprehension via graphing.
So what do we ...
3
The method you propose is already known, its basically a numerical approximation to the gradient. It is not used to train neural networks because its well... an approximation. You still need to do two forward passes to get an approximation, which introduces noise and might make the training process fail.
Using backpropagation to compute the gradient is an ...
2
There is nothing stopping you, you can setup Dense Neural Networks to have any size inputs or outputs (simple proof is to imagine a single layer NN with no activation is just a linear transform and given input dim $n$ and output dim $m$, it's just a matrix of $n$ x $m$, trivially this works though with any number of hidden layers)
The better question is ...
2
It depends. It could give you a boost or it could not.
Intuitively I would expect it to actually hurt performance if the network is initialized correctly (I think the optimizer is less of a bottleneck because they will have the same effect in both approaches).
Ideal World: We optimize the network as a whole to gain better course grained features over the ...
1
Mathematical Exploration
let $\Theta^+$ be the pseudo-inverse of $\Theta$.
Recall, that if a vector $\boldsymbol v \in R(\Theta)$ (ie in the row space) then $\boldsymbol v = \Theta^+\Theta\boldsymbol v$. That is, so long as we select a vector that is in the rowspace of $\Theta$ then we can reconstruct it with full fidelity using the pseudo inverse. Thus, ...
1
If is a truly a random number, and you could guess each of the next successive five in sequence, then you could win the lottery consistently.
This is one of the first tasks many people try to do when first learning machine learning. If the lottery is truly a random physical process with fair, i.e., balanced ping pong balls, then you cannot predict which ...
Only top voted, non community-wiki answers of a minimum length are eligible
