Which Python packages do you recommend for random search hyperparameter optimization to use? Is there any recent and good one (better than the one in scikit-learn)?


Welcome to AI.SE Enes.

I think by random search, you are referring to so-called "black-box optimization". Random search is sometimes used as a name for this, but BBO is a more common name, and might be easier to search for.

There are many BBO techniques. 'random search' is usually used to refer to a hill-climbing algorithm where you start at a random location, sample from 'adjacent' points in the input space, and then pick the best one to move to.

Scikit-optimize is fairly easy to use package, and implements some of the most reasonable algorithms (gaussian process is particularly strong on most problems). The package's website includes many good examples showing how to use the methods the package provides.

If you're not happy with those methods, then you are likely entering 'unsupported land'. There are lots of more complex, esoteric, methods, but I don't believe they are well consolidated into a single package (or even a single language). There's a nice collection on the Sahindis lab's website, but those are in a mixture of C and Fortran. There are also a huge number of algorithms submitted to recurring competitions, like this one at GECCO each year, but the implementations of these algorithms may not be available at all, or might be available only through the websites of their respective authors.

  • $\begingroup$ I mean the one that "RandomizedSearchCV" implements in Sci-kit. Is it the same as black-box optimization? $\endgroup$ Mar 8 '19 at 11:46
  • 1
    $\begingroup$ RandomizedSearchCV is a method of hyper-parameter selection, which is often (though not always) solved using black-box optimization methods. If what you want is really just RandomizedSearchCV, no newer package will implement that faster, because all it is doing is randomly sampling possible parameter values, and taking the biggest. Scikit-learn's implementation is the easiest one to use that I can see. $\endgroup$ Mar 8 '19 at 18:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.