Do representations introduce an inductive bias?

Question

I am trying to get a better understanding of inductive biases used in machine learning.

I understand inductive biases as the assumptions encoded into the learning algorithm which make it prefer one hypothesis over another.

Architectural choices are prime examples of inductive biases. The simplest example, linear regression, assumes that there is a linear relationship between input and output.

My question is whether the choice of representation forces the algorithm to prefer one hypothesis over another.

Simple example

Suppose that our problem is to predict house prices. We can select between 2 different representations:

1. House is represented as: $(x_1, x_2) \gets $ (area of house, # bedrooms)
2. House is represented as: $(x_1', x_2') \gets $ (distance from city center, # swimming pools)

Does the choice of representation introduce any additional inductive bias besides the one introduced by linear regression? I mean in both cases, we are left with mapping from $\mathbb{R}^2 \to \mathbb{R}$ with a linear function.

My view (possibly wrong)

The only way I can think of representations introducing an inductive bias is if we view the problem as following. There is an input space $X$ (the raw input) and we are interested in finding a map $X \to Y$. If we use a learner with $(x_1, x_2)$, then this amounts to finding a function:

$X \to g(X) \to h(g(X)) \to Y$

where $(x_1, x_2) = g(X)$. Now we have an inductive bias, since the function the learner must pick is the total composition $h(g(X))$ and the constraint arises since it must include $g(X)$.

Answer 1

Indeed the choice of representation can introduce additional inductive biases beyond the bias introduced by the learning algorithm itself such as your linear regression model example. The key idea here is that the features you represent shape the hypothesis space that the learner explores, and this can affect which hypotheses are more likely to be preferred or even possible within that space.

For your example, the second representation's relationship between distance from the city center or number of swimming pools and house price is intuitively complex and nonlinear, the linear model might struggle to fit the data, and you're very likely to see poor performance. Thus by choosing a particular representation of the raw data, you are implicitly introducing constraints on the types of hypotheses that are possible within the entire hypotheses space. And in general ML algos like decision trees also implicitly introduce inductive biases such as favoring more simple shallower trees.

Answer 2

Yes, representations in machine learning introduce inductive biases. Inductive bias refers to the assumptions a model makes to generalize from limited data. Representations are how data is structured or encoded for input into a model, and these representations inherently carry assumptions about the nature of the data and the relationships within it.

For example:

• Handcrafted Features: When you manually design features, you're embedding assumptions about what aspects of the data are important (e.g., edge detection in image processing or n-grams in text analysis).
• Learned Representations: In models like neural networks, the architecture and the way features are extracted (e.g., convolutional layers in CNNs) impose inductive biases. For instance, CNNs assume that nearby pixels in an image have related information.

Thus, the choice of representation influences the types of patterns the model can learn, impacting the inductive biases introduced into the learning process. A best ai video generator is explaining it at the destination