3
$\begingroup$

I am trying to make a classifier.

I am new to AI (even if I know the definition and all such a bit) , and also I have no idea of how to implement it properly by myself even if I know a bit of Python coding (in fact, I am fifteen years old !🙄🙄), but my passion for this has made me ask this (silly, probably) question.

Are there neural networks where nodes are randomly selected from among a set of nodes (in random orders and a random number of times)? I know this is from ML (or maybe deep learning, I suppose), but I have no idea how to recognize such a thing from the presently available algorithms. It will be great if you all could help me, because I am preparing to release an API for programming a model which I call the 'Insane Mind' on GitHub, and I want some help to know if my effort was fruitless.

And for reference, here's the code :

from math import *
from random import *
 
class MachineError(Exception):
    '''standard exception in the API'''
    def __init__(self, stmt):
        self.stmt = stmt
def sig(x):
    '''Sigmoid function'''
    return (exp(x) + 1)/exp(x)

class Graviton:
    def __init__(self, weight, marker):
        '''Basic unit in 'Insane Mind' algorithm
           -------------------------------------
           Graviton simply refers to a node in the algorithm.
           I call it graviton because of the fact that it applies a weight
           on the input to transform it, besides using the logistic function '''
        self.weight = weight # Weight factor of the graviton
        self.marker = marker # Marker to help in sorting
        self.input = 0 # Input to the graviton
        self.output = 0 # Output of the graviton
        self.derivative = 0 # Derivative of the output

    def process(self, input_to_machine):
        '''processes the input (a bit of this is copied from the backprop algorithm'''
        self.input = input_to_machine
        self.output = (sig(self.weight * self.input) - 1)/(self.marker + 1)
        self.derivative = (sig(self.input * self.weight) - 1) * self.input *self.output * (1- self.output) 
        return self.output
    
    def get_derivative_at_input(self):
        '''returns the derivative of the output'''
        return self.derivative

    def correct_self(self, learning_rate, error):
        '''edits the weight'''
        self.weight += -1 * error * learning_rate * self.get_derivative_at_input() * self.weight
        
class Insane_Mind:

    def __init__(self, number_of_nodes):
        '''initialiser for Insane_Mind class.
           arguments : number_of_nodes : the number of nodes you want in the model'''
        self.system = [Graviton(random(),i) for i in range(number_of_nodes)] # the actual system
        self.system_size = number_of_nodes # number of nodes , or 'system size'
        
    def  output_sys(self, input_to_sys):
        '''system output'''
        self.output = input_to_sys
        for i in range(self.system_size):
            self.output = self.system[randint(0,self.system_size - 1 )].process(self.output)
        return self.output
    
    def train(self, learning_rate, wanted):
        '''trains the system'''
        self.cloned = [] # an array to keep the sorted elements during the sorting process below
        order = [] # the array to make out the order of arranging the nodes
        temp = {} # a temporary dictionary to pick the nodes from
        for graviton in self.system:
            temp.update({str(graviton.derivative): graviton.marker})
        order = sorted(temp)
        i = 0
        error = wanted - self.output
        for value in order:
            self.cloned.append(self.system[temp[value]])
            self.cloned[i].correct_self(learning_rate, error)
            error *= self.cloned[i].derivative
            i += 1
        self.system = self.cloned

Sorry for not using that MachineError exception anywhere in my code (I will use it when I am able to deploy this API).

To tell more about this algorithm, this gives randomized outputs (as if guessing). The number of guesses vary from 1 (for a system with one node), 2 (for two nodes) and so on to an infinite number of guesses for an infinite number of nodes.

Also, I wanna try and find how much it can be of use (if this is something that has never been discovered, if it is something that can find a good place in the world of ML or Deep Learning) and where it can be used.

Thanks in advance.

Criticisms (with a clear reason) are also accepted.

$\endgroup$
  • $\begingroup$ Hi and welcome! Can you please put your main question in the title? Btw, have a look at ai.stackexchange.com/help/on-topic to know more about our site. $\endgroup$ – nbro Aug 29 at 15:58
  • $\begingroup$ @nbro, my main doubt is : is my proposed ML model already present ? $\endgroup$ – Spectre Aug 29 at 15:59
  • $\begingroup$ Are you talking about Dropout $\endgroup$ – Swakshar Deb Aug 29 at 16:05
  • $\begingroup$ Yes, maybe you're looking for something like dropout. $\endgroup$ – nbro Aug 29 at 16:05
  • 2
    $\begingroup$ It may be my fault, but I am not able to understand what you want to build and what the benefit of this. Why guessing random nodes? $\endgroup$ – Swakshar Deb Aug 29 at 16:24
2
$\begingroup$

It is difficult to prove a negative, but I do not think there is any classifier (neural network or otherwise) that fully matches to your idea.

I suspect that you will not be able to take the idea of random connections and loops at run time, and make a useful classifier out of it. That's not to say the idea is completely without merit, sometimes it is good to explore blue sky ideas and just see what happens. However, I think it might be a frustrating excercise to build anything on top of your idea without some basic foundation work first. I recommend that you look into the theory and implementation of logistic regression as a starting point, which is a good stepping stone to understanding neural networks.

There are some neural network components and architectures that make use of random behaviour at the activation level:

  • Dropout. This is a method used during training which zeroes outputs from randomly selected neurons. It often gives an effective boost to neural network stability (acting to prevent overfitting to input data) and can improve accuracy of classifiers too due to behaving similarly to having multiple simpler classifiers.

  • Boltzmann machines, and restricted Boltzmann machines (RBMs) output 0 or 1 randomly from each "neuron" unit, with the probability decided by sum of inputs. They are used to create generative models, not classifiers though. Another difference is that the randomness is applied both during training and during inference, whilst dropout is most often applied to augment training. Early on in the days of deep learning, RBMs were used to pre-train layers in a deep neural network. This was effective, but other simpler methods were discovered later and are nowadays preferred in most cases.

  • A variant of dropout call called Monte Carlo dropout is used at inference time. This can be used to measure uncertainty in a model's individual predictions, which is otherwise hard to obtain.

  • Although not quite as freeform as your random connections on a per neuron basis. If you applied dropout to a recurrent neural network, that might be quite close to your idea, because the existence of loops between neurons in each time step would be random. This could be applied in language modelling and classifiers for sequence data. The same motivations apply here as for dropout in simpler feed forward classifiers - it can in theory make a classifier more robust against noise in the inputs and more accurate.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Dropout can also be applied at inference time, which is known as Monte Carlo (MC) Dropout. $\endgroup$ – nbro Aug 30 at 1:05
  • $\begingroup$ Thanks , @NeilSlater , for your inclination to answer. So, basically, what you say is that my idea here is somewhat new (although there seems to be vague references to some previously discovered machines), but still there are chances that this machine may overfit to the data it is fed with and may be not that meaningful (so it will be better if I try exploring it), right ? $\endgroup$ – Spectre Aug 30 at 6:18
  • $\begingroup$ @NeilSlater, my model uses randomness during training as well as during the tests. $\endgroup$ – Spectre Aug 30 at 7:17
  • 1
    $\begingroup$ @Spectre: The specific combination of randomness with connections appears to be new. I don't expect it to work though, as your idea is basically to build something different without a strong reference to theory. This is a bit like trying to add a random reagent that you like the name of to a well-established scientific chemical reaction: You will make something, but it is very likely that the utility will be low. $\endgroup$ – Neil Slater Aug 30 at 10:22
  • 1
    $\begingroup$ There might be some utility to randomising connections as you are doing, but you will want to start with a stronger theoretical base to build it, you won't "discover a machine", because that's not how R&D works. None of the existing ML designs were discovered purely by experimenting with design ideas, they were all motivated by at least some underlying theory. There are elements of trial and experiment, and it is useful to be creative when searching, but as I said in the answer, you will be frustrated in your progress if all you do is generate ideas for processing, build and test them. $\endgroup$ – Neil Slater Aug 30 at 10:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.