# What is the difference between a learning algorithm and a hypothesis?

What's the distinction between a learning algorithm $$A$$ and a hypothesis $$f$$? I'm looking for a few concrete examples, if possible. From what I understand, one way to vary the hypothesis $$f$$ would be to change the parameter values, maybe even the hyperparameter values of, say, a decision tree.

How can we vary A though? Would decision tree and random forest be considered two different learning algorithms? Would a shallow neural network (that ends up learning a linear function) and a linear regression model, both of which use gradient descent to learn parameters, be considered different learning algorithms?

Also, aside from what I've mentioned, are there any other ways of varying the hypothesis $$f$$?

• Hi. Please, in general, ask one question per post. You're asking too many questions, even though they are somehow related. I've provided an answer only to the question in the title. – nbro Nov 26 '19 at 16:11

A hypothesis is a statement that suggests an as yet unproven explanation of a relationship between two or more phenomena that you intend to test. An agronomist thinks that more nitrogen on canola will always increase the crop output $$Harvest = f(N)$$, or a meteorologist thinks he can show that the path of a hurricane over the ocean can be determined by knowledge of the sea temperature and the wind speed at an altitude of 1000 feet one minute before. $$D(t,0) = f(T(t-1,1000),S(t-1,1000)$$ Both hypotheses are pegs on which later steps are based; testing follows with a conclusion whether the hypothesis can be rejected or not.

Changing a hypothesis can be simply adding or subtracting arguments to the function or changing the nature of the relationship such as the acceleration of the wind as opposed to its velocity.

A "learning" algorithm describes how the parameters of a numeric model are changed in accordance with the delta rule, that is what the learning rate is and whether momentum is to be applied.

Random Forest and Decision Tree are "classification" algorithms. They are clearly stepwise processes that proceed towards the goal of a model, but they start by specifying the shape that the model will take and place boundaries on what values the parameters may take.

Both learning and classification algorithms specify a priori what shape the model will take and by doing so limit its relevance to particular problems.

• I think that this answer is not correct or at least imprecise. Have you looked at the definition of learning algorithm and hypothesis, in the context of computational learning theory? – nbro Nov 24 '19 at 19:27
• @nbro I look forward to other superior answers, appropriately upvoted by knowledgeable readers. – Colin Beckingham Nov 24 '19 at 20:43

In computational learning theory, a learning algorithm (or learner) $$A$$ is an algorithm that chooses a hypothesis (which is a function) $$h: \mathcal{X} \rightarrow \mathcal{Y}$$, where $$\mathcal{X}$$ is the input space and $$\mathcal{Y}$$ is the target space, from the hypothesis space $$H$$.

For example, consider the task of image classification (e.g. MNIST). You can train, with gradient descent, a neural network to classify the images. In this case, gradient descent is the learner $$A$$, the space of all possible neural networks that gradient descent considers is the hypothesis space $$H$$ (so each combination of parameters of the neural network represents a specific hypothesis), $$\mathcal{X}$$ is the space of images that you want to classify, $$\mathcal{Y}$$ is the space of all possible classes and the final trained neural network is the hypothesis $$h$$ chosen by the learner $$A$$.

In the case of decision trees, a decision tree algorithm, for example, the ID3 algorithm, is the learner (or learning algorithm). The space of all trees that the learner considers is the hypothesis space.