New answers tagged math
0
I found a solution to this some time back. I have been studying function approximation (within linear regression) for some time.
Here's how I did it:
Neural Networks have been proved to be universal function approximators. So, even a single hidden layer would be sufficient to approximate a function simple as addition (Even somewhat complex functions like ...
1
Mathematical Exploration
let $\Theta^+$ be the pseudo-inverse of $\Theta$.
Recall, that if a vector $\boldsymbol v \in R(\Theta)$ (ie in the row space) then $\boldsymbol v = \Theta^+\Theta\boldsymbol v$. That is, so long as we select a vector that is in the rowspace of $\Theta$ then we can reconstruct it with full fidelity using the pseudo inverse. Thus, ...
4
There is no strict definition of suitability of an activation function for neural networks. Instead there are a number of desirable traits, and functions that don't meet them or come close enough may perform badly in general (but those functions may still work in specific cases)
If you are using gradient descent as a training method, then differentiability ...
1
Backpropagation is actually a lot easier than it is made out to be - if you have a basic understanding of calculus and the chain rule, and the single multi-variable calculus rule that to combine 2 gradient vectors, you simply add them.
This is hands down the best walk through of back prop I've found on the internet. If you are still confused after that, ...
3
It doesn't seem that it is a "proper" symbol.
I guess that $\sup$ simply refers to the supremum, that is, you want to select actions that maximize the quantity that comes to the right of $\sup$, while $\text{dist}$ is simply a proxy for any possible distance between distributions. For example, you can replace $\text{dist}$ with the Kullback-Leibler ...
2
TL;DR Here is a beautiful explanation with diagrams: source
To address:
the cell state is essentially long term memory embedding (correct me if I'm wrong)
The embedding can be long or short term and it is a vector.
To answer:
Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special ...
Top 50 recent answers are included
