3
$\begingroup$

I thought I have implemented the code (from scratch, no library) for an artificial neural network (first endeavour in the field). But I feel like I miss something very basic or obvious.

To make it short: code works for a single pair of in-/out-values but fails for sets of value pairs. I do not really understand the training process. So I want to get this issue out of the way first. The following is my improvised training (aka all that I can think of) in pseudocode.

trainingData = [{in: [0,0], out:[0]}, {in: [0,1], out:[0]}, ...];
iterations = 10000

network = graphNodesToNetwork()
links = graphLinksToNetwork()
randomiseLinkWeights(links)


while(trainingData not empty) {
  for(0<iterations) { 
     set = trainingData.pop()

     updateInput(network, set.in)

     forwardPropagate(network, links) 

     linkUpdate = backPropagate(network, links, set.out)

     updateLinks(linkUpdate, links)}
}

Is this how it is supposed to work? Do you feed in your training data set by set (while-loop)?

Edit 1: because my final comment did distract from the issue at hand.

Edit 2: less wordy, more code-y

$\endgroup$
  • $\begingroup$ Who said it pushes in the opposite direction? It may or it may not..It may also push it in a skewed direction..Also backprop with momentum exist for problem of excessive weight oscillation $\endgroup$ – DuttaA Jun 22 '18 at 7:31
  • $\begingroup$ Also to be noted how do you learn..Learning a new mathematical formulae does not necessarily null and void ur previous knowledge $\endgroup$ – DuttaA Jun 22 '18 at 7:34
  • $\begingroup$ Removed my comment for clarity. $\endgroup$ – Col. Cool Jun 22 '18 at 8:25
  • $\begingroup$ You are asking the question a bit vaguely...can you make it more mathematical in nature? $\endgroup$ – DuttaA Jun 22 '18 at 10:21
  • $\begingroup$ Do you mean in terms of what I have set up so far or do you mean in terms of describing my problem? I am afraid you mean the latter which could be an issue. Would pseudo code be acceptable? Hold on... Edited question. $\endgroup$ – Col. Cool Jun 22 '18 at 11:25
2
$\begingroup$

Your network must have something which persist like weights add bias

Your new implementation would be like this :

trainingData = [{in: [0,0], out:[0]}, {in: [0,1], out:[0]}, ...];
iterations = 10000

network = graphNodesToNetwork()
links = graphLinksToNetwork()
randomiseLinkWeights(links)
weights = []


while(trainingData not empty) {
    for(0<iterations) { 
       set = trainingData.pop()

       weights = updateInput(network, set.in)

       forwardPropagate(network, links) 

       linkUpdate = backPropagate(network, links, set.out, weights)

       updateLinks(linkUpdate, links)}
}

Inshort retain weights, and backprapogate.

$\endgroup$
1
$\begingroup$

It looks like you are training you model 10,000 times on one piece of data, and then dropping that piece and moving to the next.

This will not work: the model will become extremely good at learning one piece of data, but will then forget about it when optimizing for the next piece.

Instead, either pick one example at random in each iteration, or compute the gradient for all 4 examples and just update in that direction instead.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.