The definition I see for representational capacity is "the family of functions the learning algorithm can choose from when varying the parameters in order to reduce a training objective." (Goodfellow's Deep learning book).
However, to me this seems to be the same as the definition of the hypothesis space. Is the key difference the "in order to reduce a training objective" in that some functions may not be chosen in reducing a training objective? Or are these identical definitions.
Learning algorithms (some others too, like search) aim at generating functions that get as close as possible to the "shape" of the training data (so we can then feed values to the generated functions and get outputs like, say, a prediction).
In 2D, the "shape" may be easy to visualize. If the data in 2D seems to line up, learning algorithms generating linear/affine functions (e.g. y = ax + b), should fit fairly well. Their representational capacity extends to lines. If the data seems to form a parabola, a representational capacity bound to lines will do poorly. We then need more "capable" representations, which can cope with the quadratic terms.
So this should be what the "family of functions" Goodfellow's book refers to. The "training objective" should not be in the definition (to me), as a line fitting solution is unable to represent, say, a parabola, whatever the training objective is. However the book definition may mean a line can be fit to a parabola, strictly speaking, although it will do so very poorly.
The hypothesis space resembles the definition, perhaps (I have not checked it), but what matters is the "mindset". I would draw a parallel with prior and posterior distributions in Statistics, where we usually make some hypothesis on the shape of the distribution.
Concrete examples like linear regressions generate linear functions, as hinted by their names. Decision trees generate sets of linear functions, thus more complex and "capable". SVM generates functions depending on its kernel. The RBF kernel, for example, allows generating functions that can map quite complex, non-linear data "shapes". And arbitrary neural networks can map "arbitrary" data "shape" (easier written than actually done).
