Is a linear regression model able to figure out the relation of division among two features?

Question

I have a dataset that consists of data about students. The features are things such as passed credits, failed credits, semester among with other features. The target is the student's gpa. I believe that while the passed credits and failed credits don't have a direct correlation to the gpa (since they depend on the amount of the time students spent on the school), the values of passed credits / semester and failed credits / semester highly affect the outcome of the model. I want to know what is the best method of passing the data to a linear regression model? Should I pass passed credits / semester and failed credits / semester to the model and ignore passed credits and failed credits or vice versa? Or should I leave the data untouched and the model will figure out the relation itself?

Answer 1

What you mentioned is called feature engineering, i.e., the process of using domain knowledge to create new features or modify existing ones in a way that makes machine learning algorithms work more effectively.

Given a training set $X=\left\{x^{(i)}, y^{(i)} \right\}_{i=1}^{m}$, with $x^{(i)} \in \mathbb{R}^{n}$ being the $i$-th sample, and $y^{(i)} \in \mathbb{R}$ the outcome, if you think the ratio of two features $X_1/X_2$ is more relevant to the prediction of $y$ than $X_1$ and $X_2$ separately, then you should compute this new hand-crafted feature, and drop $X_1$ and $X_2$ (see avoiding collinearity below).

However, training a linear regression on $X_1$ and $X_2$ versus training on the new feature $\frac{X_1}{X_2}$ results in fundamentally different models. The linear regression model is defined by: $$ \hat{y}^{(i)} = \vartheta_0 + \vartheta_1 x^{(i)}_1 + \vartheta_2 x^{(i)}_2 + \ldots + \vartheta_n x^{(i)}_n $$ where $\hat{y}^{(i)}$ is the predicted value for the $i$-th sample. By contrast, if we introduce the new feature $\frac{x^{(i)}_1}{x^{(i)}_2}$, the model becomes: $$ \hat{y}^{(i)} = \vartheta_0 + \vartheta_1 \frac{x^{(i)}_1}{x^{(i)}_2} + \vartheta_3 x^{(i)}_3 + \ldots + \vartheta_n x^{(i)}_n $$ Here, the ratio $\frac{X_1}{X_2}$ is weighted by a single parameter, $\vartheta_1$, instad of $\vartheta_1$ and $\vartheta_2$ separetely. This modification changes the model’s architecture (i.e., the number of trainable parameters $\vartheta_k$) and therefore the model's performance.

Avoid collinearity

A good rule is that of dropping from your training set linearly dependent features. Suppose you have two different features, $X_1$ and $X_2$, that are linearly dependent (or collinear): $$X_2 = aX_1 + b$$ where $a, b \in \mathbb{R}$. Generally, linear regression is implemented based on normal equations, i.e., a method which gives you the best linear model that fits your data, rather than optimizing the model's weights by means of gradient-based optimization algorithms. Given a training set $X \in \mathbb{R}^{m \times n}$ and a target $y \in \mathbb{R}^{m}$, the weights $\vartheta \in \mathbb{R}^{n}$ of the linear regression model are computed as follows: $$ \vartheta = \left(X^T X \right)^{-1} X^T y$$

As you can see, an inverse matrix appears in the formula. In order to compute $A^{-1}$, the constant $\frac{1}{|A|}$ must be computed, with $|A|$ being the determinant of $A$. However, if $A$ has linear dependent rows/cols, $|A|=0$, thus $\frac{1}{|A|}$ is not defined. This also implies that $A^{-1}$ cannot be computed, and you cannot train a linear regression model under these assumptions.