3
$\begingroup$

In the paper Nonlinear Interference Mitigation via Deep Neural Networks, the the following network is illustrated.

The network structure is neural network structure

The network parameters are $\theta = \{W_1^{1},...,W_1^{l-1},W_2^{1},...,W_2^{l-1},W^{l},\alpha_1,...,\alpha_{l-1}\}$, where $W_1$ and $W_2$ are linear matrices and $\rho^{(i)}(x)=xe^{-j\alpha_i|x|^2}$ is element-wise nonlinear function ($i$ is the index of layer).

Where should I add this $\rho^{(i)}(x)$? Is it possible to learn the parameter $\alpha$? I don't think it is the same idea as activation function since it is positioned in the middle of two linear matrices... Or can it be added as embedding layer?

Improve this question
$\endgroup$
15
  • 1
    $\begingroup$ Hi and welcome to this community! Can you please tell us which paper you found these expressions in? $\endgroup$
    – nbro
    May 21, 2019 at 13:37
  • 1
    $\begingroup$ Hi thanks! It's in the field of optical communication. check arxiv.org/pdf/1710.06234.pdf Figure 1 bottom branch $\endgroup$
    – yfliuuu
    May 21, 2019 at 13:42
  • 1
    $\begingroup$ After a quick look at the relevant parts of the paper, I think that $\rho$ are the activation functions that act element-wise (so if the input to $\rho$ is a vector, the result is also a vector with the same shape as the input vector). However, the activation functions are also followed by another linear transformation, the matrices $W_2^{l}$, where $l$ is the layer. $\alpha$ should be learnable parameters (complex numbers), according to the paper. $\endgroup$
    – nbro
    May 21, 2019 at 14:01
  • 1
    $\begingroup$ Thank you very much, to my understanding, at the end of each layer there should be activation function, so how do you add another linear transformation after that? And is there a way to learn parameters in the activation function? $\endgroup$
    – yfliuuu
    May 21, 2019 at 14:05
  • 1
    $\begingroup$ It is usual the case that the non-linearity is applied after the linear transformations, but, in this architecture, it seems not to be the case: I have not read more than a few lines of the paper, so maybe they explain why they are doing this. They say that the non-linearity is a differentiable function. So, you can take the derivative of this function w.r.t. to its parameter $\alpha$. Once you have the derivative of a differentiable function w.r.t. to one of its parameters, you can use gradient descent to learn those parameters. $\endgroup$
    – nbro
    May 21, 2019 at 14:06

1 Answer 1

Reset to default
1
$\begingroup$

In general, you can learn any parameter of the network, provided you can find the partial derivative of the loss function with respect to the desired parameter. Given that $\rho$ is assumed to be differentiable (as the authors state in the paper), you can take the partial derivative of the loss function with respect to the parameter $\alpha$.

In this paper, $\rho$ is a non-linear function (that is, a function that is not linear, e.g. the sigmoid function) that applies element-wise to its input. So, if you pass a vector to this $\rho$, you will get a vector of the same shape out of it. The authors do not explicitly call it an "activation function", but $\rho$ does an analogous job of an activation function, that is, it introduces non-linearity. Furthermore, in this architecture, $\rho$ is also followed by a matrix. In general, this is not forbidden, even though it is not common.

In general, in each layer of a neural network, you can have several different learnable parameters or weights. A parameter is learnable if you can differentiate the loss function with respect to it. You can have more than one weight matrix. For example, recurrent neural networks have usually more than one weight matrix associated with each layer: one matrix is associated with the feed-forward connections and the other matrix is associated with the recurrent connections.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .