1
$\begingroup$

This question is in relation to a previous doubt of mine :

Are there neural networks where nodes are randomly selected from among a set of nodes (in random orders and a random number of times)?

I have made a bit of progress from there, refurbished my code, and got things ready.

What I intend to make is 'Insane Mind', a model which forms random linear neural networks from a set of nodes at random times ( I made out the 'linear neural network' part from a bit of Google searches).

The basic process involved is :

  1. The system forms nodes of random weights . These nodes also have the Sigmoid function (the logistic fuction : $f(x) = \frac{1}{1 + e^{-x}}$ ) , and I termed these 'Gravitons' (because of the usage of the word 'weights' in them - sorry if my terminology work seems ambiguous...😅)
  2. The input enters the system via one of the gravitons.
  3. The node processes it and either passes the output to the next node or to itself .
  4. Step 3 is repeated a certain number of times as the number of gravitons made for use.
  5. The output of the final graviton is given as the output of the whole system.

One thing I'm sure of this model is that this model can transform an input vector into an output vector.

I am not sure whether this is ambiguous or similar to previously discovered model. Plus, I'd like to know if this will be effective in any situation (I believe it will be of help in classification problems).

Note : I made this out of my imagination , which means this may be useless one way or the other, but still it seemed to work.

Here's the training algorithm I made for this model :

  1. In my Python implementation of this model, I had added a provision in the 'Graviton' class to store the derivative of the output of the graviton. Using this, the gravitons are ordered in the increasing order of the derivatives of their outputs.
  2. The first graviton is taken, and its weight is modified by the error in the output.
  3. The error is modified by the product of the graviton's output derivative and its weight after editing.
  4. Steps 2 through 3 are done for the other gravitons as well. The final error (given by the error variable ) will be the product of the derivatives, the edited weights and the error in the output.
  5. The set of gravitons thus formed is the next set subjected to this training.

For extra reference, here's the code:

  1. Insane_Mind.py :
from math import *
from random import *
 
class MachineError(Exception):
    '''standard exception in the API'''
    def __init__(self, stmt):
        self.stmt = stmt
        
def sig(x):
    '''Sigmoid function'''
    try :
        return exp(x)/(exp(x) + 1)
    except OverflowError:
        if x > 0 :
            return 1
        elif x < 0:
            return 

class Graviton:
    def __init__(self, weight, marker):
        '''Basic unit in 'Insane Mind' algorithm'''
        self.weight = weight
        self.marker = marker + 1
        self.input = 0
        self.output = 0
        self.derivative = 0

    def process(self, input_to_machine):
        '''processes the input'''
        self.input = input_to_machine
        self.output = sig(self.weight * self.input)
        self.derivative = self.input * self.output * (1- self.output) 
        return self.output
    
    def get_derivative_at_input(self):
        '''returns the derivative of the output'''
        return self.derivative

    def correct_self(self, learning_rate, error):
        '''edits the weight'''
        self.weight += -1 * error * learning_rate * self.get_derivative_at_input() * self.weight
        
class Insane_Mind_Base:
    '''Insane_Mind base class - this is what we're gonna use to build the actual machine'''
    def __init__(self, number_of_nodes):
        '''initialiser for Insane_Mind_Base class.
           arguments : number_of_nodes : the number of nodes you want'''
        self.system = [Graviton(random(),i) for i in range(number_of_nodes)] # the actual system
        self.system_size = number_of_nodes # number of nodes , or 'system size'
        
    def  output_sys(self, input_to_sys):
        '''system output'''
        self.output = input_to_sys
        for i in range(self.system_size):
            self.output = self.system[randint(0,self.system_size - 1 )].process(self.output)
        return self.output
    
    def train(self, learning_rate, wanted):
        '''trains the system'''
        self.cloned = []
        order = []
        temp = {}
        for graviton in self.system:
            temp.update({str(graviton.derivative): self.system.index(graviton)})
        order = sorted(temp)
        i = 0
        error = wanted - self.output
        for value in order:
            self.cloned.append(self.system[temp[value]])
            self.cloned[i].correct_self(learning_rate, error)
            error *= self.cloned[i].derivative * self.cloned[i].weight
            i += 1
        self.system = self.cloned

    def details(self):
        '''gets the weights of each graviton'''
        for graviton in self.system:
            print("Node : {0}, weight : {1}".format(graviton.marker , graviton.weight))

class Insane_Mind:
    
    '''Actaul Insane_Mind class'''
    def __init__(self, number_of_gravitons):
        '''initialiser'''
        self.model = Insane_Mind_Base(number_of_gravitons)
        self.size = number_of_gravitons
        
    def get(self, input):
        '''processes the input'''
        return self.model.output_sys(input)
    
    def train_model(self, lrate, inputs, outputs, epoch):
        '''train the model'''
        if len(inputs) != len(outputs):
            raise MachineError("Unequal sizes for training input and output vectors")
        epoch = str(epoch)
        if epoch.lower() == 'sys_size':
            epoch = int(self.model.system_size)
        else:
            epoch = int(epoch)
        for k in range(epoch):
            for j in range(len(inputs)):
                    val = self.model.output_sys(inputs[j])
                    self.model.train(1/val if str(lrate).lower() == 'output' else lrate, outputs[j])
    
    def details(self):
        '''details of the machine'''
        self.model.details()

  1. Insane_Mind_Test.py :
from Insane_Mind import *
from statistics import *

input_data = [3,4,3,5,4,4,3,6,5,4] # list of forces using which the coin is tossed
output_data = [1,0,0,1,1,0,0,0,1,1] # head or tails in binary form (0 = tail (= not head), 1 = head)
wanteds = output_data.copy()
model = Insane_Mind(2) # Insane Mind model
print("Before Training:")
print("----------------")
model.details() # fetches you weights of the model

def normalize(x):
    cloned = x.copy()
    meanx = mean(x)
    stdevx = stdev(x)
    for i in range(len(x)):
        cloned[i] = (cloned[i] - meanx)/stdevx
    return cloned

def random_catch(range_of_catches, sample_length):
    # sample data generator. I named it random catch as part of using it in testing whether my model 
    # ' catches the correct guess'. :)
    return [randint(range_of_catches[0], range_of_catches[1]) for i in range(sample_length)]

input_data = normalize(input_data)
output_data = normalize(output_data)

model.train_model('output', input_data, output_data, 'sys_size')
# the argument 'output' for the argument 'lrate' (learning rate) was to specify that the learning rate at # each step is the inverse of the output, and the use of 'sys_size' for the number of times to be trained
# is used to tell the machine that the required number of epochs is equal to the size of the system or 
# the number of nodes in it.

print("After Training:")
print("----------------")
model.details() # fetches you weights of the model

predictions = [model.get(i) for i in input_data]

threshold = mean(predictions)
predictions = [1 if i >= threshold else 0 for i in predictions]

print("Predicted : {0}".format(predictions))
print("Actual:{0}".format(wanteds))
mse_array = [(wanteds[j] - predictions[j])**2 for j in range(len(input_data))]
print("Mean squared error:{0}".format(mean(mse_array)))

accuracy = 0
for i in range(len(predictions)):
    if predictions[i] == wanteds[i]:
        accuracy += 1

print("Accuracy:{0}({1} out of {2} predictions correct)".format(accuracy/len(wanteds), accuracy, len(predictions)))

print("______________________________________________")

print("Random catch test")
print("-----------------")

times = int(input("No. of tests required : "))
catches = int(input("No. of catches per test"))
mse = {}
for m in range(times):
    wanted = random_catch([0,1] , catches)
    forces = random_catch([1,10], catches)
    predictions = [model.get(k) for k in forces]
    threshold = mean(predictions)
    predictions = [1 if value >= threshold else 0 for value in predictions]
    mse_array = [(wanted[j] - predictions[j])**2 for j in range(len(predictions))]
    print("Mean squared error:{0}".format(mean(mse_array)))
    mse.update({(m + 1):mean(mse_array)})
    accuracy = 0
    for i in range(len(predictions)):
        if predictions[i] == wanted[i]:
            accuracy += 1
    print("Accuracy:{0}({1} out of {2} predictions correct)".format(accuracy/len(wanteds), accuracy, len(predictions)))
    

I tried running 'Insane_Mind_Test.py', and the results I got are :

Results from 'Insane_Mind_Test.py'

The formula I used from MSE is (please correct me if I was wrong): $$ MSE = \frac{\sum_{i = 1}^n (x_i - x'_i)^2}{n}$$

where,

$$ x_i = \text{Intended output}$$ $$ x'_i = \text{Output predicted}$$ $$ n = \text{Number of outputs}$$ My main intention was to make a guess system.

Note : Here, I had to think differently. I decided to classify the forces as those yielding a head and those that yield a tail (unlike what I say in the comments in the program).

Thanks for all help in advance.

Edit: Here's the training data :

Forces         Head(1) or not head(0)[rather call it tail] 
_______        ______________________
3                  1
4                  0
3                  0
5                  1
4                  1
4                  0
3                  0
6                  0
5                  1
4                  1
$\endgroup$
  • $\begingroup$ Please do correct me if I was foolish... $\endgroup$ – Spectre Sep 1 at 11:09
  • 3
    $\begingroup$ Hi. Try to ask simpler/shorter questions (if possible). Also, if you are new to AI or ML, it would be better if you start from existing literature and knowledge rather than inventing your own. Moreover, please, put your main question in the title. Finally, read our on-topic page ai.stackexchange.com/help/on-topic. Note that programming issues are off-topic and if you have to show code in your question then maybe something is wrong. $\endgroup$ – nbro Sep 1 at 11:18
  • 2
    $\begingroup$ Please, address my other concerns. Put your main question in the title. $\endgroup$ – nbro Sep 1 at 11:43
  • 2
    $\begingroup$ In this case I think you could link your code for anyone wanting to analyse in more depth - maybe a gist or similar, so it is a snapshot related to the question. The test data is important though, as an analysis of that is likely to be part of any answer. $\endgroup$ – Neil Slater Sep 1 at 12:12
  • 2
    $\begingroup$ I attempted to word your title based on an educated guess what you most want feedback about. Please correct it if I am wrong - the title will influence the kind of answer you get in a major way. $\endgroup$ – Neil Slater Sep 1 at 12:14
0
$\begingroup$

Well, here's what I ended up with of late :

My model is completely psychological (or philosophical - no idea which to choose) and hence is theoretically possible but needn't be so mathematically (I am not sure of that part, so I leave it to you to verify that in your upcoming answers).

Therefore, there must be some changes in the terminology as well as procedures:

  1. Instead of 'gravitons' , the nodes can preferably be called 'thought centers' , taking them to be the centers of the models perception of what class the object may belong to.
  2. Editing weights randomly while training (NB:I decided to edit the training algorithm) and using them in randomly during the transformation of the input to the output - which I call 'shuffle' (number of times a weight used needn't be taken into consideration - 'freedom of thought granted to AI !') - is something which makes the model really vague when it comes to the flow of data through it, but still I prefer it, as it is basically a guesswork machine and since the guess choices can be limited by using at most 2 'thought centers' (4 choices - if you go for a single node, it will always yield $\varphi(wx)$ for a constant input $x$ [ $w$ = weight, $\varphi(x) = \frac{1}{1 + e^{-x}}$]).
  3. The code I posted here was a bit faulty, so I decided to rebuild it. The code will be posted in my GitHub profile, and if any of you would like to try it, you may download it from there (I am not good at making Python packages and I haven't made one till now, I'll be putting it up there in the master branch itself. I will tell you of the release as an edit in this answer).I built that code from scratch, integrated it into a class (Insane_Mind) and tried that on the iris dataset (not the testing part, but the training part at this time of development)

Again, if anything seems ambiguous, please tell me.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Am I correct in interpreting this answer, that your goal is to build a model that processes data semi-randomly, and it is ok that you have no evidence either way that the processing is meaningful by any objective measure? So far, you have objectively measured that the behaviour changes from epoch to epoch. This is not a classifier in a machine learning sense - defining learning as getting better at a task through experience (it won't become statistically better at guessing). That doesn't make it invalid for other purposes, but it is not at all clear what your goal is in producing this model. $\endgroup$ – Neil Slater Sep 3 at 19:07
  • $\begingroup$ If your goal is to create an abstract model of guessing behaviour or randomness generation within biological systems I think your next steps would need to provide matching theory based on understanding of real neurons in brains, or experiments with living creatures that show it is a useful model for predicting or simulating behaviour (e.g. it does better at emulating a simple creature's behaviour than a simple random number generator). $\endgroup$ – Neil Slater Sep 3 at 19:11
  • $\begingroup$ @NeilSlater, you read it right that my goal is to build a classifier. But my idea of putting randomness in classification is like this : a person without a proper understanding of how to classify a thing will surely try guessing. If the invigilator conducting the classification test tries to mislead that guy, he will surely try to alter his decision. He (the person being subjected to the test) wants to get a right guess and may alter the decisions from the false belief that the invigilator is one way or the other trying to hint him on what the correct answer is . $\endgroup$ – Spectre Sep 4 at 3:11
  • $\begingroup$ If the guy guessing has at least a small idea of how he can distinguish between different classes, he will make it out of the test with good marks some way or the other, while the guy guessing the answers will get different marks in different epochs, as long as he ceases to get a basic idea of how he can classify more effectively. $\endgroup$ – Spectre Sep 4 at 3:15
  • 1
    $\begingroup$ I don't think the weights are getting "tuned" to any meaningful state or that the model has any practical use for processing data (which is why I was asking about other goals for making the model). It might be interesting for you to see if the system gets into a stable state longer term with the same input data after e.g. 100 or 1000 epochs perhaps it always predicts 0 or 1 regardless of input. $\endgroup$ – Neil Slater Sep 4 at 6:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.