0
$\begingroup$

everyone. I am working on a nested neural network architecture. For the sake of better understanding my question, simply assume the loss is

$L = G(k’) - H(k'')$

where $G$ and $H$ are two functions we do not need to know but variable $k'$ and $k''$ stem from neural network NNs such that

$k’ = NNs(k),\quad k’’ = NNs(k’)$

so this would make the acquirement of $k''$ go through a nested architecture of the same neural networks, such that

$k'' = NNs(NNs(k))$

Apparently, to obtain output $k’’$ the same neural network gets nested once:

$k \to NNs \to k′ \to NNs \to k′′$

So my question is what is the potential issue of the nested neural network? Is there any paper that I could look into? Pytorch forum told that stale forward activations are used during a backward pass could raise issue, but I cannot find any example (the same question has been asked at Pytorch forum).

$\endgroup$

1 Answer 1

1
$\begingroup$

Isn't this just a very short recurrent neural network? Same issues apply, although they are less severe since you aren't applying as many recurrent iterations. Once you start "nesting" them more, most typical issues are vanishing and exploding gradients.

$\endgroup$
2
  • $\begingroup$ Thank you for the insightful answer. Does gradients vanishing/exploding also become severe if the input dimension increases while the recurrent iterations stay the same? $\endgroup$ Commented Nov 13, 2023 at 17:44
  • $\begingroup$ To my knowledge the input dimension doesn't affect the math on how stacked model's gradients tend to either zero or infinity (depending whether the multiplier is less or greater than one). $\endgroup$
    – NikoNyrh
    Commented Nov 15, 2023 at 11:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .