How is regression machine learning?

Question

In regression, in order to minimize an error function, a functional form of hypothesis $h$ must be decided upon, and it must be assumed (as far as I'm concerned) that $f$, the true mapping of instance space to target space, must have the same form as $h$ (if $h$ is linear, $f$ should be linear. If $h$ is sinusoidal, $f$ should be sinusoidal. Otherwise the choice of $h$ was poor).

However, doesn't this require a priori knowledge of datasets that we are wanting to let computers do on their own in the first place? I thought machine learning was letting machines do the work and have minimal input from the human. Are we not telling the machine what general form $f$ will take and letting the machine using such things as error minimization do the rest? That seems to me to forsake the whole point of machine learning. I thought we were supposed to have the machine work for us by analyzing data after providing a training set. But it seems we're doing a lot of the work for it, looking at the data too and saying "This will be linear. Find the coefficients $m, b$ that fit the data."

Answer 1

So in a sense you are correct. Using your jargon: linear regression will only "work" if the true function is approximately $y=h(x)=\beta^{T}x+\beta_0$. Advantages to using this is that its light, its convex, and all-around easy.

but for alot of larger problems, this wont work. As you said you want the machine to do the work, so this is (kinda) where deeper models come into play: You allow a learn-able featurization and classification/regression. Think about it this way, the result of your regression is most likely linearly associated with some set of features, they just may not be the ones you are interested in (you can prove this actually with any infinitely wide network :: Universal approx Thm). Unfortunately we cant use an infinitely dimensional model, so we run with these giant over-parametrized models where we hope the a good function can be described by a sub-structure (only recently are we starting to pay attention, to how these sub-structures form -- look at this paper)

But the way you bring about thinking about it is a large pit fall for many trying to move forward. Alot of ML people now of days gain success by throwing a function without alot of parameters on a big data problem, but youll see the largest advancements in the field come from a theoretical understanding of the featurization and optimization.

I hope this helped

Answer 2

Actually regression comes under the statistical analysis. As you know many business activity(decision making) relies in the previous trends that can be grabbed from the organizations transaction data. When regression is performed on those organizational data. One can understand what decision can be made. One could even simulate the different conditions when the regression line is generated and to predict the unknown cases, decision maker can pass the numerical values corresponding to the certain phenomena in the operation of the organization.

How regression is machine learning?

Let's start from the definition of machine learning.

Machine learning is an application of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning focuses on the development of computer programs that can access data and use it learn for themselves.

Source: https://www.expertsystem.com/machine-learning-definition/

As from the definition it becomes clear that machine learning is to know the inner insight about the data without being explicitly programmed. Doesn't it fills great to know about what my previous trends in the business related transaction data is trying to convey me.

Please note that, in machine learning algorithms like Regression, one is trying to built some relation between the transnational data.

So how relation between the data is being built?
Consider you are in the business of selling and buying the house and you want to predict the house price according to the latest trend. So what you have got is the data of house price and the feature of the house.

Feature : house_area, no_of_rooms
Target (what you want to predict): Price

Now, you perform regression on those data and you want to find out what would be best price for the house with the feature that is not mentioned in the latest trend's data. Suppose general regression becomes like:

price = a * hourse_area + b * no_of_rooms + some_constant

So in some sense. We're just trying to find the best fit line of the latest trend data with some variables like a, b and some_constant. Isn't it great to find such higher level of details from those trends data to know what would be the house price for non-mentioned data in the so called 'training data'.

Choosing the objective function for best mapping?
Suppose relation is sometimes is non-linear. But how my algorithms would know that. In such case, one can use Artificial-Neural Network as it can learn to hypothesize the non-linear training data too.

Note: You can learn to simulate the non linear data at: https://playground.tensorflow.org

Answer 3

It is just a statistical technique that is used in machine learning and it depends on the nature of the machine learning problem. I think you should be referred to the relation of the statistics and machine learning. These are no the same, but you can see the statistical methods in machine learning methods.

For your specific problem, there are a lot of optimization techniques in AI (not specifically in machine learning). So, I think you should scrutinize more on the problem to find the relation of machine learning, AI, and statistics in this regression example.

Answer 4

What you are asking touches upon two very different approaches to machine learning:

1. The empirical approach (many people just call this 'machine learning', and some people like to call it 'algorithmic machine learning')
2. The statistical approach (some people like to call this 'statistical machine learning')

The purely empirical approach is very goal-oriented - think discriminative models that are only used for prediction. You really only care about whether the data fits the training + test data well according to whichever metric you've selected.

The statistical approach is very process-oriented - you would want to identify the processes that generate the data, the distributions they follow, whether your results are statistically significant, etc.

Along this spectrum, most folks fall somewhere into the middle.

What you've described is closer to statistical machine learning - to practitioners of the other approach, regression only means that you are trying to predict for a continuous target variable (whereas classification would be for a discrete target variable). Then you might poke around the data a bit, fiddle around with features & hyperparameters, and try out a lot of different regression algorithms, going from OLS, SVMs, nearest neighbour regressors, random forests, gradient boosted trees, and maybe even RNNs, etc. In the extreme case, a purist of this approach wouldn't care about the statistics or whatever the underlying distributions are at all, but only care if the final regression gives good results in practice.

While there are clear risks with this approach (especially when the underlying assumptions of the models fall apart), it can give good results, especially when the practitioner is a good coder and can try out lots of possibilities very quickly, and even produce novel algorithms. The fact is that maths does sometimes lag development of other fields - Fourier analysis for example, and deep neural networks.

Another very approximate analogy would be science vs engineering.