I have a CNN architecture for CIFAR-10 dataset which is as follows:
Convolutions: 64, 64, pool
Fully Connected Layers: 256, 256, 10
Batch size: 60
Loss: Categorical Cross-Entropy
When I train this model, training and testing accuracy along with loss has a very jittery behavior and does not converge properly.
Is the defined architecture correct? Should I have a max-pooling layer after every convolution layer?