Let's say there's an optimization problem of 10 elements.

If I try to optimize it, I'll get to some result. (***)

My question is, if somehow I start optimizing while having 3 elements already solved with the almost the ground truth (AKA almost best) solution, will the optimization of the other 7 elements reach better or the same results, compared to (***) ?

Grouping the features of the concept class being learned can be used used to reduce the time required to learn the concept because it lowers the computation time of each sample and epoch. That's done almost always when recognizing complex movements or with other concept classes where the number of dimensions in the model is high. It is not a universal solution and care must be taken to configure the approach correctly.

If there is a local minimum that is directly along a particular axis in the loss surface which is not the global minimum and that axis happens to be the axis corresponding to the first feature the practitioner unwittingly chooses to optimize. The result will be that the other nine are learned immediately, yet the result is not the optimal. That's an unlikely scenario but only one of many scenarios where the technique can trouble the learning objective.

Also, the interrelation between features of the concept class can be complex such that learning some of the features may cause the unlearning of others. Since a larger loop must be placed around the entire process so that the accuracy of the features learned earlier are further tested until all the factors affecting loss are adjusted to minimize it, the process can take longer in some cases.

It is not clear whether the question uses the term Element as a synonym for Feature, but let's assume so.

In that case, the optimization of ten features, as opposed to optimization of three features, will converge more slowly.

If ... I start optimizing while having [three features] already solved with the almost the ground truth solution, will the optimization of the other [seven features] reach better or the same results, compared to [optimizing all ten together]?

The answers are maybe and no.

  • The accuracy and reliability of the convergence toward ground truth (a formalized objective used to guide the optimization process) when split into groups of three and seven features may be better or worse than when left as a group of ten.
  • Except in rare cases, the results will not be the same. The likelihood of identical results is so low it may never happen in the world in the next century except when conditions are arranged solely to cause it to occur.

Why then do may approaches group the dimensions of the result and converge on groups of axes, then another, then another, and back again to the first, iterating until the convergence goal is reached? This approach is used to reduce the time and computing resources used to reach the optimal. As the problem complexity increases, the use of groupings in this way is more common.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.