From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and weights as their constants, like Fourier sums and other orthogonal basis functions.
How, then, can one neural network layer use activation function, like a sigmoid, and another one like the output using softmax? How do we know a linear combination of sigmoids and something else can still build that function no matter what? To me, it's like saying a function is approximated using sine functions with $N$ different $k$ values and then also randomly a few Hermite polynomials are thrown in as well. In this case, Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).
This question highlights some misconceptions I have about activation functions, perhaps, and I'd like to know where I'm going wrong here.