It seems to me that, right now, the key to making a good Machine Learning model is in choosing the right combination of hyper-parameters.
Firstly: Am I right in saying, if a model is able to tune it's own hyper-parameters, we have in some sense achieved a general intelligence system? Or a glimpse of an actual artificial intelligence system?
I feel the answer lies in what one means by " tune it's own hyper-parameters". If it means to be able to reach Bayesian levels of performance on that task then theoretically, after the tuning, the model is able to perform at par or better than humans and so it seems the answer would be yes.
Secondly: I understand that hyper-parameter tuning is done intuitively. But there are a set of general directions that is discernable looking at results. Here I am talking about a heuristic approach to perfect a learning model. Consider an example: Say I hardcode a model to, while training, observe gradient values. If the gradient is too large or the cost is highly oscillatory, then restart training with a smaller learning rate. Then obtain metrics on a test set. If it is poor, then again restart training with regularisation or increased regularisation. It can also observe various plot behaviours, etc.
The point is maybe not every trick up a researcher's sleeve can be hardcoded. But a decent level of basic tuning can be done.
Thirdly: Let us say, we have a reinforcement learning system on top of a supervised learning system. That is an RL network sets some hyper-parameters. The action then is to train with these hyper-parameters. The reward would be the accuracy on the test set.
Is it possible that such a system could solve the problem of hyper-parameter tuning?