# Can you learn parameters in nonlinear function?

In the paper Nonlinear Interference Mitigation via Deep Neural Networks, the the following network is illustrated.

The network structure is The network parameters are $$\theta = \{W_1^{1},...,W_1^{l-1},W_2^{1},...,W_2^{l-1},W^{l},\alpha_1,...,\alpha_{l-1}\}$$, where $$W_1$$ and $$W_2$$ are linear matrices and $$\rho^{(i)}(x)=xe^{-j\alpha_i|x|^2}$$ is element-wise nonlinear function ($$i$$ is the index of layer).

Where should I add this $$\rho^{(i)}(x)$$? Is it possible to learn the parameter $$\alpha$$? I don't think it is the same idea as activation function since it is positioned in the middle of two linear matrices... Or can it be added as embedding layer?

• Hi and welcome to this community! Can you please tell us which paper you found these expressions in? – nbro May 21 at 13:37
• Hi thanks! It's in the field of optical communication. check arxiv.org/pdf/1710.06234.pdf Figure 1 bottom branch – Yifan Liu May 21 at 13:42
• After a quick look at the relevant parts of the paper, I think that $\rho$ are the activation functions that act element-wise (so if the input to $\rho$ is a vector, the result is also a vector with the same shape as the input vector). However, the activation functions are also followed by another linear transformation, the matrices $W_2^{l}$, where $l$ is the layer. $\alpha$ should be learnable parameters (complex numbers), according to the paper. – nbro May 21 at 14:01
• Thank you very much, to my understanding, at the end of each layer there should be activation function, so how do you add another linear transformation after that? And is there a way to learn parameters in the activation function? – Yifan Liu May 21 at 14:05
• It is usual the case that the non-linearity is applied after the linear transformations, but, in this architecture, it seems not to be the case: I have not read more than a few lines of the paper, so maybe they explain why they are doing this. They say that the non-linearity is a differentiable function. So, you can take the derivative of this function w.r.t. to its parameter $\alpha$. Once you have the derivative of a differentiable function w.r.t. to one of its parameters, you can use gradient descent to learn those parameters. – nbro May 21 at 14:06

In general, you can learn any parameter of the network, provided you can find the partial derivative of the loss function with respect to the desired parameter. Given that $$\rho$$ is assumed to be differentiable (as the authors state in the paper), you can take the partial derivative of the loss function with respect to the parameter $$\alpha$$.
In this paper, $$\rho$$ is a non-linear function (that is, a function that is not linear, e.g. the sigmoid function) that applies element-wise to its input. So, if you pass a vector to this $$\rho$$, you will get a vector of the same shape out of it. The authors do not explicitly call it an "activation function", but $$\rho$$ does an analogous job of an activation function, that is, it introduces non-linearity. Furthermore, in this architecture, $$\rho$$ is also followed by a matrix. In general, this is not forbidden, even though it is not common.