# How does size of the dataset depend on VC dimension?

This might be a little broad question, but I have been watching Caltech youtube videos on Machine Learning, and in this video prof. is trying to explain how we should interpret VC dimension in terms of what it means in layman terms, and why do we need it in practice.

The first part I think I understand, please correct me if I am wrong. VC Dimension dictates the number of effective parameters (i.e. degrees of freedom) that model has. In other words, the number of parameters model needs in order to cover all possible label combinations for the chosen dataset. Now, the second part is not clear to me. The professor is trying to answer the question: "How does knowing VC dimension of a learning algorithm affect number of samples we need for training?"

Again, I apologize if all of this may be trivial but I am new to the field and wish to learn as much as I can so I can implement better and more efficient programs in practice.

From  we know that we have the following bound between the test and train error for i.i.d samples:

$$\mathbb{P}\left(R \leqslant R_{emp} + \sqrt{\frac{d\left(\log{\left(\frac{2m}{d}\right)}+1\right)-\log{\left(\frac{\eta}{4}\right)}}{m}}\right) \geqslant 1-\eta$$

that $$R$$ is test error, $$R_{emp}$$ is training error, $$m$$ is the size of the training dataset, and $$d$$ is VC dimension of the learning algorithm. As you can see, training and test error have some relations to the size of the dataset ($$m$$) and $$d$$.

Now, in terms of PAC learnability, we want to find a (lower or upper) bound for $$m$$ such that the absolute difference between $$R$$ and $$R_{emp}$$ will be less than a given $$\epsilon$$ with a given probability of at least $$1-\eta$$. Hence, $$m$$ can be computed in terms of $$\epsilon$$, $$\eta$$, and $$d$$. For example, it can be proved () to train a binary classifier with $$\epsilon$$ difference between test and train error with the probability of at least $$1-\eta$$, we need $$O\left(\frac{d + \log\frac{1}{\eta}}{\epsilon} \right)$$ i.i.d sample data, i.e., $$m = O\left(\frac{d + \log\frac{1}{\eta}}{\epsilon}\right)$$. See more example and references here.

• Isn't it more appropriate to say VC dimension of a "model" rather than "algorithm"? You say "VC dimension of the learning algorithm". – nbro Apr 17 at 0:41
• @nbro thanks for your comment. $N$ was a mistake. It should be $m$. I said "algorithm" instead of "model" because the OP has written the professor's question like: "How does knowing VC dimension of a learning algorithm affect number of samples we need for training?" – OmG Apr 17 at 0:47
• Yes, right. I missed that. Maybe we should clarify that (in our answers). Does it ever make sense to say "VC dimension of an algorithm"? – nbro Apr 17 at 0:49
• @nbro As I know the model is a more general term, but in this context model and algorithm are using interchangably. – OmG Apr 17 at 0:56
• @OmG First of all, thank you for the answer. It provided some clarity. Second, regarding your discussion in the comments, when I said "learning algorithm" I did refer to the underlying hypothesis class. – Stefan Radonjic Apr 17 at 8:09

The VC dimension represents the capacity (the same Vapnik, the letter V from VC, calls it the "capacity") of a model (or, in general, hypotheses class), so a model with a higher VC dimension has more capacity (i.e. it can represent more functions) than a model with a lower VC dimension.

The VC dimension is typically used to provide theoretical bounds e.g. on the number of samples required for a model to achieve a certain test error with a given uncertainty or, similarly, to understand the quality of your estimation given a certain dataset.

Just to give you an idea of how the bounds look like, have a look at the theorem on page 6 (of the pdf) of the paper An overview of statistical learning theory (1999) by Vapnik.

Have also a look at this answer, where I provide more info about the VC dimension, in particular, in the context of neural networks.

• I will later update this answer to provide more relevant (to the question) bounds and maybe explain them. Alternatively, have a look at this other answer. – nbro Apr 17 at 0:11
• Thank you for the references. I am currently reading your answer on VC dimensions in the context of neural networks as we speak. – Stefan Radonjic Apr 17 at 8:11

Given a hypothesis set $$H$$, the set of all possible mappings from $$X\to Y$$ where $$X$$ is our input space and $$Y$$ are our binary mappings: $$\{-1,1\}$$, the growth function, $$\Pi_H(m)$$, is defined as the maximum number of dichotomies generated by $$H$$ on $$m$$ points. Here a dichotomy is the set of $$m$$ points in $$X$$ that represent a hypothesis. A hypothesis is just a way we classify our points. Therefore with two labels we know,

$$\Pi_H(m)\leq 2^m$$

This is just counts every possible hypothesis. The VC dimension is then the largest $$m$$ where $$\Pi_H(m)=2^m$$.

Consider a 2D perceptron, meaning our $$X$$ is $$\mathbb{R}^2$$ and our classifying hyperlane is one-dimensional: a line. The VC dimension will be 3. This is because we can shatter (correctly classify) all dichotomies for $$m=3$$. We can either have all points be the same colour, or one point be a different colour - which is $$2^3=8$$ dichotomies. You may ask what if the points we are trying to classify are collinear. This does not matter because we are concerned with resolving the dichotomies themselves, not the location of the points. We just need a set of points (wherever they may be located) that exhibits that dichotomy. In other words, we can pick the points such that they maximize the number of dichotomies we can shatter with one classifying hyperplane (a triangle): the VC dimension is a statement of the capacity of our model.

To make this clear, consider $$m=4$$. We can represent the truth table of the XOR gate as a dichotomy but this is not resolvable by the perceptron, no matter where we choose the location of the points (not linearly separable). Therefore, we can resolve a maximum of 8 dichotomies, so our VC dimension is 3. In general, the VC dimension of perceptrons is $$d+1$$ where $$d$$ is the dimension of $$X$$ and $$d-1$$ is the dimension of the classifying hyperplane.

• If the 3 points are collinear, the perceptron cannot classify them, so it's not "any" dataset of 3 points. Apart from that, your answer seems correct – nbro Apr 17 at 0:05
• Yeah, I believe that has to do with the growth function. In this case, it is $\Pi_H(m)=2^3$ and the growth function of a hypothesis set is defined to maximize the number of dichotomies so a collinear set of points wouldn't be applicable when talking about the VC dimension since that isn't classifiable with one plane - they need to be in a triangular layout. But that's a good point and I'll edit my answer to be more formal. – Archie Shahidullah Apr 17 at 0:21
• @ArchieShahidullah thanks for your answer! Maybe I did not format my question correctly, but it is not the answer I was looking for. I understand what you wanted to say, but I think other answers are more related to my actual question. Anyhow, thanks for the effort! – Stefan Radonjic Apr 17 at 8:11

Since the mathematical details have already been covered by other answers, I will try to provide an intuitive explanation. I will answer this assuming the question meant $$model$$ and not $$learning$$ $$algorithm$$.

One way to think of $$\mathcal V \mathcal C$$ dimension is that it is an indicator of the number of functions (i.e a set of functions) you can choose from to approximate your classification task over a domain. So a model (here assume neural nets, linear separators, circles, etc whose parameters can be varied) having $$\mathcal V \mathcal C$$ dimension of $$m$$ shatters all subsets of the single/multiple set of $$m$$ points it shatters.

For a learning algorithm, to select a function, which gives accuracy close to the best possible accuracy (on a classification task) from the aforementioned set of functions (shattered by your model, which means it can represent the function with $$0$$ error) it needs a certain sample size of $$m$$. For the sake of argument, let's say your set of functions (or the model shatters) contains all the possible mappings from $$\mathcal X \rightarrow \mathcal Y$$ (assume $$\mathcal X$$ contains $$n$$ points i.e finite sized, as a result number of functions possible is $$2^n$$). One of the function it will shatter is the function which performs the classification, and thus you are interested in finding it.

Any learning algorithm which sees $$m$$ number of samples can easily pick up the set of functions which agrees on these points. The number of these functions agreeing on these sampled $$m$$ points but disagreeing on the $$n-m$$ points is $$2^{(n-m)}$$. The algorithm has no way of selecting from these shortlisted functions (agreeing on $$m$$ points) the one function which is the actual classifier, hence it can only guess. Now increase the sample size and the number of functions disagreeing keeps falling and the algorithms probability of success keeps getting better and better until you see all $$n$$ points when your algorithm can identify the mapping function of the classifier exactly.

The $$\mathcal V \mathcal C$$ dimension is very similar to the above argument, except it doesn't shatter the entire domain $$\mathcal X$$ and only a part of it. This limits the models capability to approximate a classification function exactly. So your learning algorithm tries to pick a function from all the functions your model shatter, which is very close to the best possible classification function i.e there will exist a best possible (not exact) function (optimal) in your set of functions which is closest to the classification function and your learning algorithm tries to pick a function which is close to this optimal function. And thus again, as per our previous argument it will need to keep increasing the sample size to reach as close as possible to the optimal function. The exact mathematical bounds can be found in books, but the proofs are quite daunting.

• There are certain parts of this answer that are a bit misleading and confusing because you don't define "learning algorithm". For example, "Any learning algorithm which sees $m$ number of samples can easily pick up the set of functions which agrees on these points". Why? Here's another misleading or at least imprecise statement "The algorithm has no way of selecting from these shortlisted functions (agreeing on $m$ points) the one function which is the actual classifier". How do you know? There are cases where there's a unique solution and you don't have to "guess". – nbro Apr 17 at 12:41
• When you say " Now increase the sample size and the number of functions disagreeing keeps falling", it's also not clear why the "functions disagreeing keeps falling" and you should word that "functions disagreeing keeps falling" more appropriately. – nbro Apr 17 at 12:42
• @nbro Its pretty clear actually. I will adjust my answer along with OPs doubts (if some details need to be added). Otherwise it is pretty clear. – DuttaA Apr 17 at 13:21