0
$\begingroup$

From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and weights as their constants, like Fourier sums and other orthogonal basis functions.

How, then, can one neural network layer use activation function, like a sigmoid, and another one like the output using softmax? How do we know a linear combination of sigmoids and something else can still build that function no matter what? To me, it's like saying a function is approximated using sine functions with $N$ different $k$ values and then also randomly a few Hermite polynomials are thrown in as well. In this case, Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).

This question highlights some misconceptions I have about activation functions, perhaps, and I'd like to know where I'm going wrong here.

$\endgroup$
2
  • 1
    $\begingroup$ have a look at this question Neural Network composed of multiple activation functions ... this might help $\endgroup$ – Faizy Apr 15 at 14:11
  • $\begingroup$ This approximation of $x^3 + x^2 - x - 1$ might help you understand a bit better: desmos.com/calculator/cfvtjusqmq. Play around with the added functions, and try combining more than one activation function and you'll find you'll still be able to approximate the function without issue. This is what a neural network is doing, just instead of 2 dimensional space it's $n$-dimensional $\endgroup$ – Recessive May 20 at 3:05
0
$\begingroup$

First of all, it looks like you are under impression that a neural network is structured like this (example for 4 inputs and outputs):

$$ \begin{array}{rcl} y_1 & = & \text{sigmoid}(w_{11}x_1 + w_{12}x_2+w_{13}x_3+w_{14}x_4+b_1)\\ y_2 & = & \text{sigmoid}(w_{21}x_1 + w_{22}x_2+w_{23}x_3+w_{24}x_4+b_2)\\ y_3 & = & \text{softmax}(w_{31}x_1 + w_{32}x_2+w_{33}x_3+w_{34}x_4+b_3)\\ y_4 & = & \text{softmax}(w_{41}x_1 + w_{42}x_2+w_{43}x_3+w_{44}x_4+b_4)\\ \end{array} $$

If that is how you understand it, then you've missed the point of "deep" neural networks. (Sequential) deep neural network are structured in several layers:

$$ l_i^1 = \sigma^1(\sum_jw_{ij}^1x_j+b^1_i) \\ l_i^2 = \sigma^2(\sum_jw_{ij}^2l^1_j+b^2_i) \\ \ddots\\ l_i^\alpha = \sigma^\alpha(\sum_jw_{ij}^\alpha l^{\alpha-1}_j+b^\alpha_i) \\ y_i = \sigma^{out}(\sum_jw_{ij}^{out} l^\alpha_j+b^{out}_i) \\ $$

The activation function $\sigma^l$ of a single layer stays the same.

To me, it's like saying a function is approximated using sine functions with N different k values and then also randomly a few Hermite polynomials are thrown in as well.

To follow this analogy - you are mistaken by thinking that we are simultaneously trying to decompose over $\sin$ functions and hermitian polynomials. What actually happens is that we first decompose over $\sin$ functions and then decompose over hermitian polynomials. This actually can makes sense from practical point of view.

Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).

They are not, but sigmoids (or ReLUs) are not orthogonal to each other either. Orthogonality has noting to do with any of it.

$\endgroup$
3
  • $\begingroup$ I agree about the linear combination of affine functions thrown in an activation function is what constitutes a neuron at layer $l$. Are you saying that we form something of a "separate" function approximation at each layer, using a different or not different activation function? What otherwise do you mean by "we first decomponse over $sin$ functions and then decompose over hermitian polynomials? I read that as, "we could decompose with $sin$ for one layer, then decompose with hermitians the next". If that's true, why do we do that? It sounds redundant. Surely I still have it wrong. $\endgroup$ – sangstar Apr 15 at 18:16
  • $\begingroup$ From my understanding, neural networks ultimately aim to approximate a function. You can approximate any function with a combination of certain functions like the hermitian polynomials we mentioned. Each neuron can constitute a different sigmoid, and all of those sigmoids in a layer are used to calculate the activation function in the next layer for each neuron. To be honest, I come from a fairly not-super-high-level in applied mathematics, and I seem to get a sense of a lot of overcomplexity in function approximation with all these layers, but I know this is incredibly naive to say. $\endgroup$ – sangstar Apr 15 at 18:21
  • $\begingroup$ My background is basically saying that you can approximate $y = mx + b$ perfectly well with a sum of cosines and sines, or hermite polynomials, or what have you (if they form an orthogonal basis / have an inner product of 0), but in a neural network, by analogy, it sounds like the function $y = mx + b$ is first decomposed (for example -- not actually in practice) with sines and cosines, and then with hermitian polynomials. Why do we.. decompose again? What good does it do us? I know this is a painfully fundamental question, so I want to be clear my misunderstandings are made clear. $\endgroup$ – sangstar Apr 15 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.