So I wrote a convolutional neural network for a binary image classification. I have around 5300 images for each class which I thought would be enough to at least give me a good accuracy on the training cases. When the training was finished, I wrote a program to find the accuracy for each class (here's what I mean: consider class A. I ran a for loop through all the training images that belonged to class A, gave the images to my classifier and divided the number of images that my classifier classified as a class A image by 5300).For class A, the accuracy was pretty good (around 96%). But for class B, the accuracy was only 80%. Further more, I did the same thing with the testing images and found out that the accuracy for class A was 95% but for class B was 70%.
What can I do in order to improve my accuracy? I thought about augmentation but wasn't sure whether to do it for both classes or just class B.
Thanks in advance!
P.S: I think the main problem that causes this is that the two classes are really alike but I don't really know how to deal with it so any help regarding that would be appreciated!