-1
$\begingroup$

It seems to me that, right now, the key to making a good Machine Learning model is in choosing the right combination of hyper-parameters.

Firstly: Am I right in saying, if a model is able to tune it's own hyper-parameters, we have in some sense achieved a general intelligence system? Or a glimpse of an actual artificial intelligence system?

I feel the answer lies in what one means by " tune it's own hyper-parameters". If it means to be able to reach Bayesian levels of performance on that task then theoretically, after the tuning, the model is able to perform at par or better than humans and so it seems the answer would be yes.

Secondly: I understand that hyper-parameter tuning is done intuitively. But there are a set of general directions that is discernable looking at results. Here I am talking about a heuristic approach to perfect a learning model. Consider an example: Say I hardcode a model to, while training, observe gradient values. If the gradient is too large or the cost is highly oscillatory, then restart training with a smaller learning rate. Then obtain metrics on a test set. If it is poor, then again restart training with regularisation or increased regularisation. It can also observe various plot behaviours, etc.

The point is maybe not every trick up a researcher's sleeve can be hardcoded. But a decent level of basic tuning can be done.

Thirdly: Let us say, we have a reinforcement learning system on top of a supervised learning system. That is an RL network sets some hyper-parameters. The action then is to train with these hyper-parameters. The reward would be the accuracy on the test set.

Is it possible that such a system could solve the problem of hyper-parameter tuning?

Improve this question
$\endgroup$
2
  • $\begingroup$ I don't know if that's what you're asking for, but neuroevolution may be something of your interest $\endgroup$
    – olinarr
    Jun 20, 2019 at 7:03
  • $\begingroup$ It's not clear exactly what you're asking. I know that this is an old post, but, please, edit your post to ask a clear question. Also, remove the first question, which seems to be completely unrelated. $\endgroup$
    – nbro
    Jun 19, 2020 at 22:58

1 Answer 1

Reset to default
2
$\begingroup$

Such a system can and does solve the problem of hyperparameter tuning. Google's AutoML does this. Here is another example that uses a Genetic Algorithm to breed new neural network structures. AutoML has been shown to outperform humans in the rate that it improves network designs. It seems to favour Residual Network style topologies.

Improve this answer
$\endgroup$
0

Not the answer you're looking for? Browse other questions tagged .