I'm trying to learn more about AI by trying to program a neural network. First I'm trying to understand writing my own perceptron but I'm struggling to get a basic perceptron working correctly.
I've tried writing a few basic perceptrons to do very basic tasks for example trying to classify a point as above or below a line y = x.
The problem I'm facing is it seems that when training the perceptron my weights start increasing exponentially and it doesn't seem like I'm getting anywhere.
If I start with a perceptron with two weights representing an x and a y value of a point on a graph: starting weight values of 1 and 1 Using the sign function of the output being more than or equal to 0 outputs +1 and less than 0 is -1.
I'm using the training data with examples (1,2), (2,1), (3,3), (2,1). I think I'm correct in saying if the perceptron outputs the correct value then you don't need to adjust the weights, if it's incorrect then the weights can be updated using
new_weight = old_weight + ((expected_ouptut - false_output) * input)
By the first iteration I have weights of -4 and -2 respectively, the second passes, the third i.e (3,3) with weights (-4,-2) changes the weights to (53,55).
I'm fairly sure my math is correct, and it seems like the weights are increasing exponentially without making any difference to getting a working perceptron. Are there any errors I'm making with trying to design a working perceptron?
Thanks!
Sorry this may get slightly long! EDIT:
I think I may have been making mistakes doing it manually, I’ve written the code now that seems to be partly working.
The way my code works is by running test firstly with test cases, followed by the results from those test cases in the same order within the lists. Followed by the weights and finally the test case to run it from. So running in ghci:
test [(5,2)] [(-1)] (1,1) (5,2)
this works correctly, first it tests the coordinates (5,2) in the perceptron with weights (1,1) and checks to see if it is equal to -1, if it is not equal to one it will run it again and alter the weights with the alter function until it does equal -1, when this happens it then checks the perceptron again and outputs with the final tuple (5,2) which is the test case to check if it’s working. This works correctly, however when working with larger sets it doesn’t work. Next I tried
test [(1,2),(2,5),(3,1),(7,0),(6,5),(4,4)] [1,1,(-1),(-1),(-1),1] (1,1) (5,1)
where the first two lists are the test cases and the results, then the weights (1,1) and finally after running all of the test cases trying to teach the perceptron it then runs the perceptron on the final tuple (5,1) and outputs the result, it should be (-1) as it is below y = x, however it outputs 1.
Pseudocode:
testFunction
if(test cases remaining):
if calling sign on the output of the perceptron != result for test case then:
call same test case again but alter weights
else:
call testFunction again will same lists minus the first test arguments and results
else if(no more test cases, all have been tested and removed, or none specified):
call sign on the output of the perceptron for the final tuple provided with the altered weights
Code in Haskell:
module Main where
-- function to take training data, results from that data to test the perceptron
-- also takes weights and finally outputs the result of the test case
test :: [(Float,Float)] -> [Float] -> (Float,Float) -> (Float,Float) -> Float
test [] [] weights testCase = sign(perceptron testCase weights)
test (t:trainData) (r:trainResult) weights testCase
| (output /= r) = test ([t] ++ trainData) ([r] ++ trainResult) (alter weights output t r) testCase -- add t and r
| otherwise = test trainData trainResult weights testCase
where
output = sign(perceptron t weights)
-- function that computes x1*w1 + x2*w2
perceptron :: (Float,Float) -> (Float,Float) -> Float
perceptron coordinate weights = ((((fst coordinate) * (fst weights))) + ((snd coordinate) * (snd weights)))
-- function that finds the new weights by doing w1 = w1 + (expected_result - perceptron output)*x1 and similar for w2
alter :: (Float,Float) -> Float -> (Float,Float) -> Float -> (Float,Float)
alter (w1,w2) output (x1,x2) result = ((w1 + ((result - output) * x1)), (w2 + ((result - output) * x2)))
-- activation function
sign :: Float -> Float
sign n
| n >= 0 = 1
| otherwise = (-1)