I'm confused about the following issue. Let assume that we have a neural network that takes one input and two outputs. I try to visualize my model like as follows:

        / --- First stream    --- > output_1
Input --
        \ ---- Second stream  ---> output_2

I used sgd with momentum. Is there any difference between using one optimizer for both streams and using two optimizers for each stream? In other words, if i use one optimizer, can one stream optimization process affect another stream? If it can, How can it be possible?

  • 1
    $\begingroup$ Yes you can. You can interpret the two streams as two entirely different NN, that only join at the final. So if you're using softmax, you can then apply a softmax across all nodes, in both streams at the final layer, then do normal cross entropy loss to get the gradient. I'm not sure how well this would work, it is a bit of a strange way of doing this. You might have better luck just keeping the NN whole, and having 2 normal outputs that you apply a single optimiser to $\endgroup$ – Recessive Dec 18 '19 at 0:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.