Matrix Dimension for Linear regression coefficients

Question

While reading about least squares implementation for machine learning I came across this passage in the following two photos:

Perhaps I’m misinterpreting the meaning of beta but if X^T has dimension 1 x p and beta has dimension p x K, then hat{Y} would have dimension 1 x K and would be a row vector. According to the text, vectors are assumed column vectors unless otherwise noted.

Can someone provide clarification?

Edit: the matrix notation in this text confuses me. The pages preceding the above passage read:

Should the matrix referenced not have dimensions p x N, assuming a p-vector is a column vector with p elements?

Note: The passage is taken from “Elements of Statistical Learning” by Hastie, Tibshirani, & Friedman.

Answer 1

This is a good example of what happens when you take text out of context.

The passage that was added in the edited question makes a difference, but it's not quite sufficient, and it doesn't help that the notation is all over the place. I found the textbook and the relevant passages (Section 2.3, p. 10-11). Here is a quick attempt at an explanation.

The authors call X a variable (either scalar or vector) with components X_j, but later in the paragraph they refer to X_j as a variable (which I think is the correct notation). Instead of "variable" with "components", think of X as a set of p variables (in the normal sense of a variable, such as temperature, the price of shares, etc.). You can put multiple variables x_i (i = 1...p) in a vector X and call each instance of X an observation. In other words, an observation is a set of measured values for all variables x_i.

Assume you have made N observations. Arrange your observations in a matrix with N rows and p columns, where each row represents a single observation (an instance of X).

Now also assume that you are trying to find the relation between your input variables x_i and a different set of variables y_k, called dependent or response variables. In general, the variables x_i and y_k are measured, and the model is simply trying to extract the relation between the input and dependent variables so you can then predict the latter from the former.

As a side note, observations are usually denoted with superscripts (x⁽ⁿ⁾) and variables with subscripts (x_i) so there is no confusion about which is which. x_i⁽ⁿ⁾ is the n^th observed (measured) value of variable x_i.

In your case, you have a single dependent variable y and p input variables x_i (i = 1...p). Assuming that their relation is linear (note: in many cases this assumption is not justified), we can assign weights ("importance") to each variable and try to find out those weights from measurements. In your case, the weights are denoted with beta_i (so the "importance" of variable x_i is beta_i; note the same subscript).

If you have N observations of your dependent variable y, then you can arrange them in a column, just like the observations for your input variables. Note that we still have a single dependent variable, so essentially y is a scalar, but the N observations of that scalar form an N-dimensional column vector.

Now multiply the N x p input matrix by the p-dimensional column vector of weights beta. What you get is a column vector of N predicted values for y (\hat{y}). The difference of the N-dimensional column vector of predicted values \hat{y} and the N-dimensional column vector y of measured values is the error (which is minimised with the least squares method).

If you have q dependent variables y_i (i = 1...q), each of them would generally have its own set of weights beta. In other words, the relation between the input variables and each y_i will be different. The dependent variables can also be arranged in a matrix (just like the input matrix), and its dimensionality will be N x q, where q. In that case, the beta matrix will not be a single N-dimensional column vector but an N x q matrix. Each column in the beta matrix gives the relation between the input variables and the q^th dependent variable y_q. In that respect, a single observation of all variables y_i will be a row vector in the y matrix.

I hope that this clarifies things up a bit. In summary, the explanation in the textbook is correct, but the notation is hard to follow and at times plain misleading (as in the case of variable and component at the beginning of the section). Honestly, you can get a much more intuitive explanation of linear regression from the Wikipedia article.

Answer 2

I have used W as beta and Y_pred as Y_hat.

Apparently, as far as conventions go both feature vector X and weight vector W are assumed as column vectors. Although this is not important in your case but when we use Neural Nets, this is particularly important, and a weight vector for a layer is given by p*N_n where p is the number of features being input from the previous layer and N_n is the number of nodes in the layer. Check a NN structure and you will understand what I am saying.

As far as the special case in your current question, the book is correct, although comments say Y_predicted should be vector, it is not, since we are doing dot product between X and W (although it is represented in a vectorised form).

Also how can W have dimensions p*K (assuming it is not a NN), it means multiple solutions to the same problem? Instead it is X which will have dimensions p*K and W will have p*1. It means you have K training examples with p features.

If its a NN with K nodes then dimensions for X like p*N and W as p*K is totally possible. The only downside of this notation is that you have to again take a transpose of the result.

Here is the explanation of the notation used:

Notation for NN