New answers tagged activation-function
3
Short answer: Generally, you don't need to do softmax if you don't need probabilities. And using raw logits leads to more numerically stable code.
Long answer: First of all, the inputs of the softmax layer are called logits.
During evaluation, if you are only interested in the highest-probability class, then you can do argmax(vec) on the logits. If you want ...
1
Rosenblatt was probably discussing a specific architecture, for which there are many. However, for general purpose feed-forward back-propagation ANNs used for function aproximation and classification analysis, you can use whatever activation functions you want on the input-side, hidden layers, and output-side. Examples are identity, logistic, tanh, ...
1
I commonly use softmax for all 2-class or k-class problems, basically, because I always like to have an output node for each class. For sigmoid, i.e., logistic, you cannot estimate MSE for each sample using the relationship
$E_i = \sum_c^C (y_c - \hat{y}_c)^2$,
where $C$ is the number of classes, $y_c$ is 0 or 1 for true class membership, and $\hat{y}_c$ is ...
2
Your analogy is correct, except it is not really an "analogy". Sin is an activation function - in past works (before modern deep learning boom) it was rather standard to see it listed as a possible activator.
So your expression $\sigma(x) = A\sin ax + B \sin bx + D \sin ex$ is of a neural network with one 3-neuron layer and a single output linear ...
1
Sigmoid is used for binary cases and softmax is its generalized version for multiple classes. But, essentially what they do is over exaggerate the distances between the various values.
If you have values on a unit sphere, apply sigmoid or softmax on those values would lead to the points going to the poles of the sphere.
0
Ok. Here is an analogy for you. The equation for a neuron is wx + b, which is equivalent to a straight line. If we don't apply non-linearity we will be stuck with a straight line forever. So, this type of network won't be even able to model points in a unit circle randomly distributed.
What does non-linearity do? If you look the graphs for x to the power 2, ...
0
First of all, it looks like you are under impression that a neural network is structured like this (example for 4 inputs and outputs):
$$ \begin{array}{rcl}
y_1 & = & \text{sigmoid}(w_{11}x_1 + w_{12}x_2+w_{13}x_3+w_{14}x_4+b_1)\\
y_2 & = & \text{sigmoid}(w_{21}x_1 + w_{22}x_2+w_{23}x_3+w_{24}x_4+b_2)\\
y_3 & = & \text{softmax}(w_{31}...
1
two hidden layers each comprising two neurons
From your description it looks like that you only have 6 parameters for your inner layer (2x2 weight matrix + 2 biases). The whole network should be easy to interpret: you've got two 13-dimensional weight vectors $\vec{w}_1,\vec{w}_2$ that are dot-multiplied with the inputs, plus two biases $b$ and activation $\...
Top 50 recent answers are included
