# How can I model any structure for a neural network?

Hello I am currently doing research on the effect of altering a neural network's structure. Particularly I am investigating what affect would putting a random DAG (directed acyclic graph) in the hidden layer of a network instead of a usual fully connected bipartite graph.

For instance my neural network would look something like this:

Basically I want the ability to create any structure in my hidden layer as long as it remains a DAG [add any edge between any node regardless of layers]. I have tried creating my own library to do so but it proved to be much more tedious than anticipated therefore I am looking for ways to do this on existing libraries such as Keras, pytorch, or tensorflow.

• You may want to expand the diagram to make it clearer how arbitrary the connections are allowed to be. The answer about residual networks might not be enough for you if you want to work at the level of individual neurons Jun 29 at 10:56

You seem to be seeking an implementation of a Residual Neural Network (https://en.m.wikipedia.org/wiki/Residual_neural_network), or ResNET for short. If you want some premade networks, the module tf.keras.applications.resnet from tensorflow (do check TF's documentation) might help you.

Mentioned frameworks don't restrict you with linear layers sequence, you could do any acyclic sequence. I.e. very popular resnet architecture based on skip-connections that jumps over the layers.

I.e. simple example on pytorch

import torch
from torch import nn
import torch.nn.functional as F

class Custom(nn.Module):
def __init__(self):
super().__init__()
self.a = nn.Parameter(torch.tensor(1.))
self.b = nn.Parameter(torch.tensor(2.))
self.c = nn.Parameter(torch.tensor(3.))

def forward(self, x):
x1 = F.relu(self.a * x)
# take note, we skip
out = F.relu(self.b * x1 + self.c*x)
return out

model = Custom()
print('before', model.a)
# You could do pretty much the same training
x = torch.tensor([2])
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2)
prediction = model(x)
loss = criterion(prediction, torch.tensor([20.]))
loss.backward()
optimizer.step()
print('after', model.a)


With the existing frameworks PyTorch, Tensorflow you can easily implement this functionality, by keeping some of the intermediate computations inside forward or call method and passing them as an input to the given layer. For example:

x[i] = layer[i-1](x[i - 1])
...
x[j] = layer[j-1](x[j - 1] + x[i]) # resnet-like skip connection
or
x[j] = layer[j-1](concat(x[j - 1], x[i])) # densenet-like skip connection


However, if you are asking whether, there is some more educated way, or a constructor of this functionality, as far as I know, it is not implemented in these frameworks.

In case you want to have a computational graph of some arbitrary DAG structure, you will need to create some structures inside you NN class in order to know, where to keep activations in order to pass them further as skip connection and where to sum/concatenate them with the output of some other layer.