How to perform back-propagation in Decoupled Neural Interfaces?

Question

I am attempting to create a fully decoupled feed-forward neural network by using decoupled neural interfaces (DNIs) as explained in the paper Decoupled Neural Interfaces using Synthetic Gradients (2017) by Max Jaderberg et al. As in the paper, the DNI is able to produce a synthetic error gradient that reflects the error with respect the output:

$$ \frac{\partial L}{\partial h_{i}} $$

I can then use this to update the current layer's parameters by multiplying by the parameters to get the loss with respect to the parameters:

$$ \frac{\partial L}{\partial \theta}=\frac{\partial L}{\partial h_{i}} * \frac{\partial h_{i}}{\partial \theta} $$

In the paper, the layer's model is then updated based on the next layer sending the true error backwards.

My question is, given that I am able to calculate the error with respect to the current output, how do I use this to calculate the loss with respect to the previous layer's output?

Answer 1

I kid you not when I tell you I just started with decoupled networks this morning. But the issue is terribly worded by most computer scientists. The way a decoupled neural interface works is not by sending the error backwards, but by storing it in a form of learned error inside a separate network. So not only do we have our nodes and weights, but also the Synths. The Synths modify the weight tables using past error. So instead of passing error back like the article suggests, it actually is passing it forward in a kind of "hey don't do this." What's really fun is that to you never have to mix up your thinking to reverse a network for backprop if you're using a decoupled network. The amazing part is that it is actually easier to code this decoupled network, than it is to write a "simple" ANN, even better still is that it functions a lot like an LSTM network without all the programming mumbo-jumbo. You can see for yourself two different forms of the same network from here.

The code from the demo in the repository is from https://towardsdatascience.com/only-numpy-implementing-and-comparing-combination-of-google-brains-decoupled-neural-interfaces-6712e758c1af it was a fundamental resource for me, and it may be for you. It also helps you with loss, or cost.