3
$\begingroup$

Main question

Is there some way we can leverage general knowledge of how certain hyperparameters affect performance, to very rapidly get some sort of estimate for how good a given architecture could be?

Elaboration

I'm working on a handwritten character recognition problem using CNNs. I want to try out a few different architectures (mostly at random) to iterate towards something which might work. The problem is that one run takes a really long time.

So what's a way to quickly verify if a given architecture is promising? And let me elaborate on what I've tried:

  • Just try it once. Yeah but maybe I chose some bad hyperparameter combination and actually that architecture was going to be the ground breaker.
  • Do Bayesian optimisation. That's still really slow. From examples and trials, I've seen that it takes quite some time for convergence. And besides, I'm not trying to optimise yet, I just want to check if there's any potential.
$\endgroup$
  • 1
    $\begingroup$ I am afraid that there are (still) no quick ways to evaluate the architecture of a NN. If there were, they probably would already be very famous and you would have found them with a quick search on the web, but I may be unaware of them (and I've not searched for them on the web after this question), even though I am familiar with certain hyper-parameter optimization techniques (like grid search, Bayesian optimization, RL, etc). $\endgroup$ – nbro Jan 15 at 0:13
  • $\begingroup$ @nbro while looking over the Bayesian charts on papers, articles and my own work I notice that something near enough to the best value is often found in the first few iterations.the rest seems like just a bunch of noise while getting marginal improvements on the best trial (actually if you know of a counter example I'd love to see it - I'm very new to the field so I won't be surprised if you have one). $\endgroup$ – Alexander Soare Jan 15 at 7:47
  • $\begingroup$ The other thing that inspires the thought is that if you randomly sample from a continuous function, if it's not a crazily dynamic one, you're likely to get an idea of it's full range after just a few samples. So let's say I have a one parameter function and I know that within my domain of interest it only has n significant turning points, then if I'm lucky I might get an idea of what it could look like in just n samples. And with some strategy I might not have to go much further than that even when I'm unlucky. $\endgroup$ – Alexander Soare Jan 15 at 7:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.