I've built a classifier to classify a dataset consisting of n samples and four classes of data. To this end, I've used pretrained VGG-19, pretrained Alexnet and even lenet (with cross entropy loss). However, I just changed the softmax layer's architecture and placed just four neurons for that (because my dataset includes just four classes). Since the dataset classes has striking resemblance to each other, this classifier were unable to classify them and I was forced to use another methods. During the training section, after some epochs, loss decreased from approximately 7 to approximately 1.2, but there was no changes in accuracy and it was frozen on 25% (random precision). In best epoches accuracy just reached near 27% but it was completely unstable.
How is it justifiable? If Loss reduction means model improvement, why accuracy doesn't increase? How is it possible to loss decreases near 6 points (approximately from 7 to 1) but nothing happen to accuracy at all?