While we train a CNN model we often experiment with the number of filters, the number of convolutional layers, FC layers, filter size, sometimes stride, activation function, etc. More often than not after training the model once, it is just a trial & error process.
Is there a way that helps me to architect my model fundamentally before training?
Once I train model, how do I know which among these variables (number of filters, size, number of convolutional layers, FC layers) should be changed - increased or decreased?
P.S. This question assumes that data is sufficient in volume and annotated properly and still accuracy is not up to the mark. So, I've ruled out the possibility of non-architectural flaws for the question.