3
$\begingroup$

So, my question is a bit theoretical. I have been trying to implement a perceptron based classifier with outputs 1 and 0 depending on the category. I have used 2 methods: The example by Example learning method and Batch learning method. I also have defined another method which will measure accuracy according to the formulae number_of_samples_classified_correctly/total_number_of_samples(I'm not sure this should be the correct definition for accuracy and you are welcome to suggest a better measure). Now there are a few confusions i'm facing. Firstly, the accuracy of example by example learning is different from batch learning by 2%. Also the best accuracy achieved in both cases is depending on the slopes. So where exactly is the mistake?(Batch learning algorithm=error*input_vector( where error can be 1,-1 or 0 ) summed over all input vectors and then added to weights).

For initial slope[1,-1] giving an accuracy of 88% example by example learning

For initial slope[1,-1] giving an accuracy of 88% batch learning

For initial slope[1,1] giving an accuracy of 84% example by example learning

For initial slope[1,1] giving an accuracy of 88% batch learning

  • For initial slope[1,-1] giving an accuracy of 88% example by example learning
  • For initial slope[1,-1] giving an accuracy of 88% batch learning
  • For initial slope[1,1] giving an accuracy of 84% example by example learning
  • For initial slope[1,1] giving an accuracy of 86% batch learning
$\endgroup$
  • $\begingroup$ I would kindly request you to post this same question here Cross Validated,as well. $\endgroup$ – quintumnia Oct 31 '17 at 19:38
  • $\begingroup$ @quintumnia do i have to write the question again or is there an easier way to export the question? $\endgroup$ – DuttaA Nov 4 '17 at 5:52
  • $\begingroup$ The community mod.can migrate it there,as long as you accept that. $\endgroup$ – quintumnia Nov 4 '17 at 19:43
2
$\begingroup$

In Brief:

re-train your dataset. I believe where you get lower accuracy scores, your model has not converged to the final state. duplicate your dataset multiple times and create a bigger one, then train your model with it.

In Detail:

number_of_samples_classified_correctly/total_number_of_samples(I'm not sure this should be the correct definition for accuracy and you are welcome to suggest a better measure)

This is a valid accuracy metric. In fact if the value is acc, then 1-acc is called misclassification error. So your metric is good unless you have some class imbalance, where you need to use other metrics such as Cohen's Kappa score.

the accuracy of example by example learning is different from batch learning by 2%. Also the best accuracy achieved in both cases is depending on the slopes.

I strongly believe both strange results happen because the number of your instances (i.e. examples) is low, or let's say your learning rate is small. As you know Perceptron algorithm corrects the weights of the decision hyper-plane by delta learning rule: it reads each instance, calculates the error (in case of binary classification {-1,0,1}) and updates the weights by c.x.E where:

  • c is learning constant/rate/step
  • x is the data instance
  • E is the error

Thus the weights (or slopes) change a little bit every time. There is no guarantee that your model reaches its final states after all the instances are given to it. Since when you start training with different initial weights, or change the order of instances given to your model, the distance or from or moving speed toward the final state changes. So as I mentioned above, re-train your model and let it converge to the final state OR increase your learning step (learning rate) . I believe all the 4 mentioned-above-accuracies will be the same then; I mean not only you should have the same model for different initial weights, but also for batch and online training.

Please update your post with new findings and question my claims if those were false.

$\endgroup$
  • $\begingroup$ Actually i compensated the low learning rate by a large number of epochs..with 100000 epochs for learning rate=0.001...also i took the best accuracy among the weights and plotted it..also somehow the accuracy measure is not working because for batch learning the final classifying line is going above both the data sets i.e for both the data sets value is negative for all training vectors yet the accuracy is still 88%....also i am not strictly using delta rule for batch learning, i am taking the error to be either [1 ,0, -1]..maybe now you understand where the problem lies? $\endgroup$ – DuttaA Nov 4 '17 at 5:48
  • $\begingroup$ Also as for higher learning rates it is causing overflow above a learning rate of 0.3 with very high weights...basically it is having an amplifying effect $\endgroup$ – DuttaA Nov 4 '17 at 10:33
  • $\begingroup$ @DuttaA higher learning rate will surely result in amplifying effect. increasing training epochs instead of increasing learning rate was a good idea... plus, why are you including [0] class? if you are using one decision boundary, then it means your dataset must have had two classes. for a single instance of dataset, being located on the decision boundary doesn't mean "0" label beside 1 and -1. if your dataset has 3 classes, use multiple classification approaches (either OvO or OvA) which means you have more than one decision boundary. maybe the odd results are caused by this interpretation. $\endgroup$ – Alireza Nov 4 '17 at 12:45
  • $\begingroup$ my dataset only has 2 classes...actually i am new to this and i am studying from a book which has used 1 and 0 as to whether in class 1 or class 2 and defined error as target-classification result hence the error might be 1, 0 or -1. $\endgroup$ – DuttaA Nov 4 '17 at 17:46
0
$\begingroup$

After running the code few more times i am thinking the problem lies in the discrete nature of error. In batch learning we are deciding the error as [0, 1, -1] depending on the input and the input whereas in the measuring of accuracy we are not making consideration of how big the error is i.e. whether the line is classifying the 2 classes by a very small margin (just classifying it) or classifying it by a good margin (comfortably classifying it)..both will give the same error since we are not considering the distance of the points from the line. Also, on choosing the initial weights we are selecting a point in the error vs weight function (which is not continuous) and if our choice of weights are inappropriate, the classifier weights is either getting trapped in a local minima or taking a very large number of epochs to reach the best weight (there is a very little probability that the classifier weights hit the right value after which it is easily able to achieve global minima). This is validated by the fact that on choosing a suitable weight we are getting the best accuracy possible within very few epochs but on choosing a different weight it doesn't reach the best accuracy but comes close to it after a large number of epochs.

As for the example by example learning method, in this type of training the initial weights will determine the local minima the classifier weights get trapped in and its impossible to get out of it even with a large number of epochs.

The 2 ways i think to solve the problem is by either using the Delta rule or by reinforcement learning (although i am not sure whether it can be applied to such discrete error functions) where the algorithm can see that even if it doesn't get an immediate gain by moving in a direction, but finally get a much more gain(thus will be able to get out of local minima) i.e exploitation vs exploration.

Any further insights, suggestions or corrections are welcome.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.