Is there some way we can leverage general knowledge of how certain hyperparameters affect performance, to very rapidly get some sort of estimate for how good a given architecture could be?
I'm working on a handwritten character recognition problem using CNNs. I want to try out a few different architectures (mostly at random) to iterate towards something which might work. The problem is that one run takes a really long time.
So what's a way to quickly verify if a given architecture is promising? And let me elaborate on what I've tried:
- Just try it once. Yeah but maybe I chose some bad hyperparameter combination and actually that architecture was going to be the ground breaker.
- Do Bayesian optimisation. That's still really slow. From examples and trials, I've seen that it takes quite some time for convergence. And besides, I'm not trying to optimise yet, I just want to check if there's any potential.