Hello I am currently doing research on the effect of altering a neural network's structure. Particularly I am investigating what affect would putting a random DAG (directed acyclic graph) in the hidden layer of a network instead of a usual fully connected bipartite graph.

For instance my neural network would look something like this: enter image description here

Basically I want the ability to create any structure in my hidden layer as long as it remains a DAG [add any edge between any node regardless of layers]. I have tried creating my own library to do so but it proved to be much more tedious than anticipated therefore I am looking for ways to do this on existing libraries such as Keras, pytorch, or tensorflow.


You seem to be seeking an implementation of a Residual Neural Network (https://en.m.wikipedia.org/wiki/Residual_neural_network), or ResNET for short. If you want some premade networks, the module tf.keras.applications.resnet from tensorflow (do check TF's documentation) might help you.


Mentioned frameworks don't restrict you with linear layers sequence, you could do any acyclic sequence. I.e. very popular resnet architecture based on skip-connections that jumps over the layers.

I.e. simple example on pytorch

import torch
from torch import nn
import torch.nn.functional as F

class Custom(nn.Module):
    def __init__(self):
        self.a = nn.Parameter(torch.tensor(1.))
        self.b = nn.Parameter(torch.tensor(2.))
        self.c = nn.Parameter(torch.tensor(3.))

    def forward(self, x):
        x1 = F.relu(self.a * x)
        # take note, we skip
        out = F.relu(self.b * x1 + self.c*x)
        return out

model = Custom()
print('before', model.a)
# You could do pretty much the same training
x = torch.tensor([2])
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2)
prediction = model(x)
loss = criterion(prediction, torch.tensor([20.]))
print('after', model.a) 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.