Why can a neural network use more than one activation function?

Question

From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and weights as their constants, like Fourier sums and other orthogonal basis functions.

How, then, can one neural network layer use activation function, like a sigmoid, and another one like the output using softmax? How do we know a linear combination of sigmoids and something else can still build that function no matter what? To me, it's like saying a function is approximated using sine functions with $N$ different $k$ values and then also randomly a few Hermite polynomials are thrown in as well. In this case, Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).

This question highlights some misconceptions I have about activation functions, perhaps, and I'd like to know where I'm going wrong here.

Answer 1

First of all, it looks like you are under impression that a neural network is structured like this (example for 4 inputs and outputs):

$$ \begin{array}{rcl} y_1 & = & \text{sigmoid}(w_{11}x_1 + w_{12}x_2+w_{13}x_3+w_{14}x_4+b_1)\\ y_2 & = & \text{sigmoid}(w_{21}x_1 + w_{22}x_2+w_{23}x_3+w_{24}x_4+b_2)\\ y_3 & = & \text{softmax}(w_{31}x_1 + w_{32}x_2+w_{33}x_3+w_{34}x_4+b_3)\\ y_4 & = & \text{softmax}(w_{41}x_1 + w_{42}x_2+w_{43}x_3+w_{44}x_4+b_4)\\ \end{array} $$

If that is how you understand it, then you've missed the point of "deep" neural networks. (Sequential) deep neural network are structured in several layers:

$$ l_i^1 = \sigma^1(\sum_jw_{ij}^1x_j+b^1_i) \\ l_i^2 = \sigma^2(\sum_jw_{ij}^2l^1_j+b^2_i) \\ \ddots\\ l_i^\alpha = \sigma^\alpha(\sum_jw_{ij}^\alpha l^{\alpha-1}_j+b^\alpha_i) \\ y_i = \sigma^{out}(\sum_jw_{ij}^{out} l^\alpha_j+b^{out}_i) \\ $$

The activation function $\sigma^l$ of a single layer stays the same.

To me, it's like saying a function is approximated using sine functions with N different k values and then also randomly a few Hermite polynomials are thrown in as well.

To follow this analogy - you are mistaken by thinking that we are simultaneously trying to decompose over $\sin$ functions and hermitian polynomials. What actually happens is that we first decompose over $\sin$ functions and then decompose over hermitian polynomials. This actually can makes sense from practical point of view.

Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).

They are not, but sigmoids (or ReLUs) are not orthogonal to each other either. Orthogonality has noting to do with any of it.