Can multiple activation functions be replaced with a single activation function?

Question

I'm just started to learn deep learning and I have a question about this neural network:

I think $h_1$, $h_j$ and $h_n$ are perceptrons. So, if they are perceptrons, all of them will have an activation function.

I'm wondering if it is possible to have only one activation function, and sum the output to all of the perceptrons, and pass that sum to that activation function. And the output of this activation function will be $y$.

I will have this network, where $H1$, $Hj$ and $Hn$ don't have activation function:

The input for the activation function will be the sum of the outputs of $H1$, $Hj$ and $Hn$ without been processed by an activation function.

Is that possible (or is it a good idea)?

Answer 1

TL;DR: This is possible but removing the activations will decrease the expressivity of the network because it will become mathematically equivalent to a single neuron.

Mathematical Explanation

The outputs of your intermediate neurons (in the absence of activation functions) are now:

$\text{(1)}\quad H_i(x) = \sum_{j=1}^nw_{ij}\cdot x_j+b_i$

You are then summing each $H_i$:

$\text{(2)}\quad\mathcal{H(x)}=\sum_{i=1}^nH_i(x)$

You then pass $\mathcal{H(x)}$ to some activation say $g$:

$\text{(3)}\quad\hat y = g(\mathcal{H(x)})$

The trouble is that the inner term $\mathcal{H(x)}$ mathematically reduces to a single linear operation on $x$. Proof:

1. $\mathcal{H(x)}=\sum_{i=1}^nH_i(x)$. Substituting in (1):
2. $\mathcal{H(x)}=\sum_{i=1}^n((\sum_{j=1}^nw_{ij}\cdot x_j)+b_i)$. This can be re-aranged :
3. $\mathcal{H(x)}=(\sum_{j=1}^n(\sum_{i=1}^nw_{ij})\cdot x_j)+\sum_{i=1}^nb_i$. But this reduces to:
4. $\mathcal{H(x)}=\sum_{j=1}^n\tilde w_{j}\cdot x_j+\tilde b$. Where $\tilde w_{j},\tilde b$ are scalars.

Thus, without the non-linear activations (3) mathematically reduces to a single neuron.

Answer 2

Of course, it is possible, but why do you want to do this?

Let's think about it. Imagine your weights for that layer is a matrix full of ones, so, if you have no bias, then your output of that layer would be the sum of all values in the $h_1$, $h_j$, $h_n$ neurons, right? So, it is possible to sum all the values together, give the output to an activation function, and then you'll have your output.