# Can feature engineering change the selection of the model according to the minimum description length?

The definition of MDL according to these slides is:

The minimum description length (MDL) criteria in machine learning says that the best description of the data is given by the model which compresses it the best. Put another way, learning a model for the data or predicting it is about capturing the regularities in the data and any regularity in the data can be used to compress it. Thus, the more we can compress a data, the more we have learnt about it and the better we can predict it.

MDL is also connected to Occam’s Razor used in machine learning which states that "other things being equal, a simpler explanation is better than a more complex one." In MDL, the simplicity (or rather complexity) of a model is interpreted as the length of the code obtained when that model is used to compress the data.

To put it in short according to MDL principle, we prefer predictors with relatively smaller description length (i.e can be described within a certain length) for a given description language (this definition is without delving into exact technical details as it is not necessary to the question).

Since MDL is dependent on the description language we use, can we say feature engineering can cause a change in the selection of the predictor? For example as this picture shows:

To me, it seems that, in the first picture, we will require a longer description length predictor in Cartesian coordinates, as compared to a predictor in polar co-ordinates (just a single discerning radius needs to be specified). So, to me, it seems feature engineering changed the selection of the predictor to a relatively simple one (in the sense that it will have a shorter description length). Thus feature engineering has changed the description length required for out predictor. Did I make any wrong assumptions? If so why?

• You're completely right. For points in cartesian coordinates, your final model would require at least 3 parameters a,b,c since the equation of a circle is: $(x-a)^2+(y-b)^2=c^2$ while in polar coordinates you final model would require only one parameter since a vertical line $p=a$ is enough to separate the two classes. – Edoardo Guerriero Apr 10 '20 at 0:39

To make this clearer, imagine you call me up on the phone, and we're both looking at your left hand image. You say to me 'Yeah, you just draw a line through it at $$p=a$$'. The natural question for me to ask is 'What's p?'. If you have to tell me what $$p$$ is, then it's part of the description length.
A circular model for your lefthand image does have a MDL for its decision boundary of something like $$(x−a)^2+(y−b)^2=c$$. However, the feature transformation you've selected has a description length of $$p=(x-a)^2 + (y-b)^2$$. It should be clear that the description lengths for the linear model through $$p$$ and the circular model through $$x$$ and $$y$$ are identical.