I'm just started to learn deep learning and I have a question about this neural network:

enter image description here

I think $h_1$, $h_j$ and $h_n$ are perceptrons. So, if they are perceptrons, all of them will have an activation function.

I'm wondering if it is possible to have only one activation function, and sum the output to all of the perceptrons, and pass that sum to that activation function. And the output of this activation function will be $y$.

I will have this network, where $H1$, $Hj$ and $Hn$ don't have activation function: enter image description here

The input for the activation function will be the sum of the outputs of $H1$, $Hj$ and $Hn$ without been processed by an activation function.

Is that possible (or is it a good idea)?

  • $\begingroup$ Your question is a little unclear can you clarify: "I'm wondering if it is possible to have only one, and sum the output to all of the perceptrons, and pass that sum to an activation function." I think I understand but I want to make sure I truly understand your question. Do you mean is it possible to add another hidden layer with only one neuron? $\endgroup$ – respectful Oct 9 '19 at 16:35
  • $\begingroup$ you mean one activation function for every neuron ? $\endgroup$ – basilisk Oct 9 '19 at 17:15
  • $\begingroup$ I have updated my question. $\endgroup$ – VansFannel Oct 9 '19 at 17:19

TL;DR: This is possible but removing the activations will decrease the expressivity of the network because it will become mathematically equivalent to a single neuron.

Mathematical Explanation

The outputs of your intermediate neurons (in the absence of activation functions) are now:

$\text{(1)}\quad H_i(x) = \sum_{j=1}^nw_{ij}\cdot x_j+b_i$

You are then summing each $H_i$:


You then pass $\mathcal{H(x)}$ to some activation say $g$:

$\text{(3)}\quad\hat y = g(\mathcal{H(x)})$

The trouble is that the inner term $\mathcal{H(x)}$ mathematically reduces to a single linear operation on $x$. Proof:

  1. $\mathcal{H(x)}=\sum_{i=1}^nH_i(x)$. Substituting in (1):
  2. $\mathcal{H(x)}=\sum_{i=1}^n((\sum_{j=1}^nw_{ij}\cdot x_j)+b_i)$. This can be re-aranged :
  3. $\mathcal{H(x)}=(\sum_{j=1}^n(\sum_{i=1}^nw_{ij})\cdot x_j)+\sum_{i=1}^nb_i$. But this reduces to:
  4. $\mathcal{H(x)}=\sum_{j=1}^n\tilde w_{j}\cdot x_j+\tilde b$. Where $\tilde w_{j},\tilde b$ are scalars.

Thus, without the non-linear activations (3) mathematically reduces to a single neuron.


Of course, it is possible, but why do you want to do this?

Let's think about it. Imagine your weights for that layer is a matrix full of ones, so, if you have no bias, then your output of that layer would be the sum of all values in the $h_1$, $h_j$, $h_n$ neurons, right? So, it is possible to sum all the values together, give the output to an activation function, and then you'll have your output.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.