3

Section 5.2 Error Decomposition of the book Understanding Machine Learning: From Theory to Algorithms (2014) gives a description of the approximation error and estimation error in the context of empirical risk minimization (ERM) and, in particular, in the context of the bias-complexity tradeoff (which is strictly related to the bias-variance tradeoff). Error/...


2

A hypothesis space/class is the set of functions that the learning algorithm considers when picking one function to minimize some risk/loss functional. The capacity of a hypothesis space is a number or bound that quantifies the size (or richness) of the hypothesis space, i.e. the number (and type) of functions that can be represented by the hypothesis space. ...


1

Error Estimation is a subject with a long history. The test-set method is only one way to estimate generalization error. Others include resubstitution, cross-validation, bootstrap, posterior-probability estimators, and bolstered estimators. These and more are reviewed, for instance, in the book: Braga-Neto and Dougherty, "Error Estimation for Pattern ...


1

Trying to explain the idea of VC to some of my colleagues I've discovered quite an intuitive way of laying out the basic idea. Without going through lots of math and notation as I've done in my other answer. Imagine a following game between two players $\alpha$ and $\beta$ : First, player $\alpha$ plots $d=4$ points on a piece of paper. She may place the ...


1

Shattered set. First we need a concept of a shattered set. I'll work from a shattered set example in Wikipedia adjusting it to your notation. The statement that $\mathcal{H}$ shatters $C$ means that for every subset $A \subset C$ there is a set $B\in\mathcal{H}$ such that $B$ "separates" $A$ from $C \backslash A$. Writing this formally: $$\text{...


1

Model/network design has multiple guidelines, a basic one is: The solving capacity of the network should be larger than the possibility space of the problem to be solved. Solving capacity (learning capacity) of a network (dense usually) can be calculated as the product of number of neurons in all layers, for example: Input shape: 10 values Network shape: [...


1

The most obvious way more classes increase the network size it the output layer, but I don't believe there is a rule of thumb for the size of the entire network. As I understand it, there is no clear answer how big a network needs to be to achieve a certain performance with regard to the number of layers compared to the number of classes. This is a very ...


1

Some of the books that you mention are often used as reference books in introductory courses to machine learning or artificial intelligence. For example, if I remember correctly, in my introductory course to machine learning, the professor suggested the book Pattern Recognition And Machine Learning (2006) by Bishop, although we never used it during the ...


1

Pattern Recognition And Machine Learning is a great theoretical book. I don't know anything better on standard ML. I read several pages from it myself and all my colleagues researchers suggest to look there if you are not sure about some concepts. The 2 problems with it are that it's huge and it doesn't cover almost all deep learning models known for today. ...


1

Yes, you're interpreting the $\max$ there wrongly. In your second formula $$ \operatorname{Regret}_{T}(\mathcal{H})=\max _{h^{\star} \in \mathcal{H}} \operatorname{Regret}_{T}\left(h^{\star}\right) \label{1}\tag{1} $$ The sign $=$ means "is defined as", so maybe the following notation is less confusing $$ \operatorname{Regret}_{T}(\mathcal{H}) \...


1

For standard NNs, their extrapolation behavior an important aspect for financial applications cannot be controlled due to complex functional forms typically involved. Neural Networks with Asymptotics Control discuss how they overcome this significant limitation and develop a new type of neural networks that incorporate large-value asymptotics, when known, ...


Only top voted, non community-wiki answers of a minimum length are eligible