Are there neural networks where nodes are randomly selected from among a set of nodes (in random orders and a random number of times)?

Question

I am trying to make a classifier.

I am new to AI (even if I know the definition and all such a bit) , and also I have no idea of how to implement it properly by myself even if I know a bit of Python coding (in fact, I am fifteen years old !🙄🙄), but my passion for this has made me ask this (silly, probably) question.

Are there neural networks where nodes are randomly selected from among a set of nodes (in random orders and a random number of times)? I know this is from ML (or maybe deep learning, I suppose), but I have no idea how to recognize such a thing from the presently available algorithms. It will be great if you all could help me, because I am preparing to release an API for programming a model which I call the 'Insane Mind' on GitHub, and I want some help to know if my effort was fruitless.

And for reference, here's the code :

from math import *
from random import *
 
class MachineError(Exception):
    '''standard exception in the API'''
    def __init__(self, stmt):
        self.stmt = stmt
def sig(x):
    '''Sigmoid function'''
    return (exp(x) + 1)/exp(x)

class Graviton:
    def __init__(self, weight, marker):
        '''Basic unit in 'Insane Mind' algorithm
           -------------------------------------
           Graviton simply refers to a node in the algorithm.
           I call it graviton because of the fact that it applies a weight
           on the input to transform it, besides using the logistic function '''
        self.weight = weight # Weight factor of the graviton
        self.marker = marker # Marker to help in sorting
        self.input = 0 # Input to the graviton
        self.output = 0 # Output of the graviton
        self.derivative = 0 # Derivative of the output

    def process(self, input_to_machine):
        '''processes the input (a bit of this is copied from the backprop algorithm'''
        self.input = input_to_machine
        self.output = (sig(self.weight * self.input) - 1)/(self.marker + 1)
        self.derivative = (sig(self.input * self.weight) - 1) * self.input *self.output * (1- self.output) 
        return self.output
    
    def get_derivative_at_input(self):
        '''returns the derivative of the output'''
        return self.derivative

    def correct_self(self, learning_rate, error):
        '''edits the weight'''
        self.weight += -1 * error * learning_rate * self.get_derivative_at_input() * self.weight
        
class Insane_Mind:

    def __init__(self, number_of_nodes):
        '''initialiser for Insane_Mind class.
           arguments : number_of_nodes : the number of nodes you want in the model'''
        self.system = [Graviton(random(),i) for i in range(number_of_nodes)] # the actual system
        self.system_size = number_of_nodes # number of nodes , or 'system size'
        
    def  output_sys(self, input_to_sys):
        '''system output'''
        self.output = input_to_sys
        for i in range(self.system_size):
            self.output = self.system[randint(0,self.system_size - 1 )].process(self.output)
        return self.output
    
    def train(self, learning_rate, wanted):
        '''trains the system'''
        self.cloned = [] # an array to keep the sorted elements during the sorting process below
        order = [] # the array to make out the order of arranging the nodes
        temp = {} # a temporary dictionary to pick the nodes from
        for graviton in self.system:
            temp.update({str(graviton.derivative): graviton.marker})
        order = sorted(temp)
        i = 0
        error = wanted - self.output
        for value in order:
            self.cloned.append(self.system[temp[value]])
            self.cloned[i].correct_self(learning_rate, error)
            error *= self.cloned[i].derivative
            i += 1
        self.system = self.cloned

Sorry for not using that MachineError exception anywhere in my code (I will use it when I am able to deploy this API).

To tell more about this algorithm, this gives randomized outputs (as if guessing). The number of guesses vary from 1 (for a system with one node), 2 (for two nodes) and so on to an infinite number of guesses for an infinite number of nodes.

Also, I wanna try and find how much it can be of use (if this is something that has never been discovered, if it is something that can find a good place in the world of ML or Deep Learning) and where it can be used.

Thanks in advance.

Criticisms (with a clear reason) are also accepted.

Answer 1

It is difficult to prove a negative, but I do not think there is any classifier (neural network or otherwise) that fully matches to your idea.

I suspect that you will not be able to take the idea of random connections and loops at run time, and make a useful classifier out of it. That's not to say the idea is completely without merit, sometimes it is good to explore blue sky ideas and just see what happens. However, I think it might be a frustrating excercise to build anything on top of your idea without some basic foundation work first. I recommend that you look into the theory and implementation of logistic regression as a starting point, which is a good stepping stone to understanding neural networks.

There are some neural network components and architectures that make use of random behaviour at the activation level:

• Dropout. This is a method used during training which zeroes outputs from randomly selected neurons. It often gives an effective boost to neural network stability (acting to prevent overfitting to input data) and can improve accuracy of classifiers too due to behaving similarly to having multiple simpler classifiers.

• Boltzmann machines, and restricted Boltzmann machines (RBMs) output 0 or 1 randomly from each "neuron" unit, with the probability decided by sum of inputs. They are used to create generative models, not classifiers though. Another difference is that the randomness is applied both during training and during inference, whilst dropout is most often applied to augment training. Early on in the days of deep learning, RBMs were used to pre-train layers in a deep neural network. This was effective, but other simpler methods were discovered later and are nowadays preferred in most cases.

• A variant of dropout call called Monte Carlo dropout is used at inference time. This can be used to measure uncertainty in a model's individual predictions, which is otherwise hard to obtain.

• Although not quite as freeform as your random connections on a per neuron basis. If you applied dropout to a recurrent neural network, that might be quite close to your idea, because the existence of loops between neurons in each time step would be random. This could be applied in language modelling and classifiers for sequence data. The same motivations apply here as for dropout in simpler feed forward classifiers - it can in theory make a classifier more robust against noise in the inputs and more accurate.