# Can multiple activation functions be replaced with a single activation function?

I think $$h_1$$, $$h_j$$ and $$h_n$$ are perceptrons. So, if they are perceptrons, all of them will have an activation function.

I'm wondering if it is possible to have only one activation function, and sum the output to all of the perceptrons, and pass that sum to that activation function. And the output of this activation function will be $$y$$.

I will have this network, where $$H1$$, $$Hj$$ and $$Hn$$ don't have activation function:

The input for the activation function will be the sum of the outputs of $$H1$$, $$Hj$$ and $$Hn$$ without been processed by an activation function.

Is that possible (or is it a good idea)?

• Your question is a little unclear can you clarify: "I'm wondering if it is possible to have only one, and sum the output to all of the perceptrons, and pass that sum to an activation function." I think I understand but I want to make sure I truly understand your question. Do you mean is it possible to add another hidden layer with only one neuron? – respectful Oct 9 '19 at 16:35
• you mean one activation function for every neuron ? – basilisk Oct 9 '19 at 17:15
• I have updated my question. – VansFannel Oct 9 '19 at 17:19

TL;DR: This is possible but removing the activations will decrease the expressivity of the network because it will become mathematically equivalent to a single neuron.

Mathematical Explanation

The outputs of your intermediate neurons (in the absence of activation functions) are now:

$$\text{(1)}\quad H_i(x) = \sum_{j=1}^nw_{ij}\cdot x_j+b_i$$

You are then summing each $$H_i$$:

$$\text{(2)}\quad\mathcal{H(x)}=\sum_{i=1}^nH_i(x)$$

You then pass $$\mathcal{H(x)}$$ to some activation say $$g$$:

$$\text{(3)}\quad\hat y = g(\mathcal{H(x)})$$

The trouble is that the inner term $$\mathcal{H(x)}$$ mathematically reduces to a single linear operation on $$x$$. Proof:

1. $$\mathcal{H(x)}=\sum_{i=1}^nH_i(x)$$. Substituting in (1):
2. $$\mathcal{H(x)}=\sum_{i=1}^n((\sum_{j=1}^nw_{ij}\cdot x_j)+b_i)$$. This can be re-aranged :
3. $$\mathcal{H(x)}=(\sum_{j=1}^n(\sum_{i=1}^nw_{ij})\cdot x_j)+\sum_{i=1}^nb_i$$. But this reduces to:
4. $$\mathcal{H(x)}=\sum_{j=1}^n\tilde w_{j}\cdot x_j+\tilde b$$. Where $$\tilde w_{j},\tilde b$$ are scalars.

Thus, without the non-linear activations (3) mathematically reduces to a single neuron.

• I am not sure if it is a good idea to say that it is not advisable, because it depends on the problem that one needs to solve. I think it is more accurate to say that it decreases the powerfulness or expressivity of the neural network. – nbro Oct 10 '19 at 2:19
• @nbro I thought about saying that but wasn't sure if it would be helpful. Seeing as you think so I'll edit. After editing it is indeed a much better explanation. – respectful Oct 10 '19 at 3:32

Of course, it is possible, but why do you want to do this?

Let's think about it. Imagine your weights for that layer is a matrix full of ones, so, if you have no bias, then your output of that layer would be the sum of all values in the $$h_1$$, $$h_j$$, $$h_n$$ neurons, right? So, it is possible to sum all the values together, give the output to an activation function, and then you'll have your output.