3
$\begingroup$

While reading about least squares implementation for machine learning I came across this passage in the following two photos: photo1 photo 2

Perhaps I’m misinterpreting the meaning of beta but if X^T has dimension 1 x p and beta has dimension p x K, then hat{Y} would have dimension 1 x K and would be a row vector. According to the text, vectors are assumed column vectors unless otherwise noted.

Can someone provide clarification?

Edit: the matrix notation in this text confuses me. The pages preceding the above passage read:

3rd 4th

Should the matrix referenced not have dimensions p x N, assuming a p-vector is a column vector with p elements?

Note: The passage is taken from “Elements of Statistical Learning” by Hastie, Tibshirani, & Friedman.

$\endgroup$
  • $\begingroup$ The text says Here we are modeling a single output, so K = 1. In that case, \hat{Y} would be a 1 x 1 vector. You could argue that this is not technically a scalar, but this is a minor point. If you are modelling more than one output, K would be greater than 1 and \hat{Y} would be a 1 x K row vector where K is the number of outputs. $\endgroup$ – cantordust Jul 3 '18 at 3:10
  • $\begingroup$ @cantordust thank you for your reply. Is this a typical convention, to have Y as a row vector? I only ask because I would expect it to be written as Y^T. But I’m just beginning my reading and haven’t seen applications enough to verify. In a linear algebra context, one would expect Y to be a column vector, but perhaps convention is different here? $\endgroup$ – Hanzy Jul 3 '18 at 3:13
  • $\begingroup$ I suspect that the reason why they have omitted the ^T is that it doesn't matter for a scalar (a happens to be the same as a^T when a is a 1 x 1 vector). But in principle you are right that it should be Y^T. This is kind of sloppy writing. They have also missed the ^ in the last sentence - that should be \hat{\beta}. $\endgroup$ – cantordust Jul 3 '18 at 3:20
  • $\begingroup$ @cantordust I noticed that as well and was left wondering if that beta was different from \hat{\beta} but assumed it was the same. There is another section on the preceding page regarding their matrix notation which I found confusing from a linear algebra context. I’ll perhaps update the question to include it. $\endgroup$ – Hanzy Jul 3 '18 at 3:24
  • $\begingroup$ Sadly, modern textbooks are rushed out the door without being proofread properly... $\endgroup$ – cantordust Jul 3 '18 at 3:26
1
$\begingroup$

This is a good example of what happens when you take text out of context.

The passage that was added in the edited question makes a difference, but it's not quite sufficient, and it doesn't help that the notation is all over the place. I found the textbook and the relevant passages (Section 2.3, p. 10-11). Here is a quick attempt at an explanation.

The authors call X a variable (either scalar or vector) with components Xj, but later in the paragraph they refer to Xj as a variable (which I think is the correct notation). Instead of "variable" with "components", think of X as a set of p variables (in the normal sense of a variable, such as temperature, the price of shares, etc.). You can put multiple variables xi (i = 1...p) in a vector X and call each instance of X an observation. In other words, an observation is a set of measured values for all variables xi.

Assume you have made N observations. Arrange your observations in a matrix with N rows and p columns, where each row represents a single observation (an instance of X).

Now also assume that you are trying to find the relation between your input variables xi and a different set of variables yk, called dependent or response variables. In general, the variables xi and yk are measured, and the model is simply trying to extract the relation between the input and dependent variables so you can then predict the latter from the former.

As a side note, observations are usually denoted with superscripts (x(n)) and variables with subscripts (xi) so there is no confusion about which is which. xi(n) is the nth observed (measured) value of variable xi.

In your case, you have a single dependent variable y and p input variables xi (i = 1...p). Assuming that their relation is linear (note: in many cases this assumption is not justified), we can assign weights ("importance") to each variable and try to find out those weights from measurements. In your case, the weights are denoted with betai (so the "importance" of variable xi is betai; note the same subscript).

If you have N observations of your dependent variable y, then you can arrange them in a column, just like the observations for your input variables. Note that we still have a single dependent variable, so essentially y is a scalar, but the N observations of that scalar form an N-dimensional column vector.

Now multiply the N x p input matrix by the p-dimensional column vector of weights beta. What you get is a column vector of N predicted values for y (\hat{y}). The difference of the N-dimensional column vector of predicted values \hat{y} and the N-dimensional column vector y of measured values is the error (which is minimised with the least squares method).

If you have q dependent variables yi (i = 1...q), each of them would generally have its own set of weights beta. In other words, the relation between the input variables and each yi will be different. The dependent variables can also be arranged in a matrix (just like the input matrix), and its dimensionality will be N x q, where q. In that case, the beta matrix will not be a single N-dimensional column vector but an N x q matrix. Each column in the beta matrix gives the relation between the input variables and the qth dependent variable yq. In that respect, a single observation of all variables yi will be a row vector in the y matrix.

I hope that this clarifies things up a bit. In summary, the explanation in the textbook is correct, but the notation is hard to follow and at times plain misleading (as in the case of variable and component at the beginning of the section). Honestly, you can get a much more intuitive explanation of linear regression from the Wikipedia article.

$\endgroup$
  • $\begingroup$ Your explanation will only confuse the OP if he tries to implement an NN or a LR model by himself..Since you just explained the confusing book instead of giving a total overview $\endgroup$ – DuttaA Jul 3 '18 at 14:21
1
$\begingroup$

I have used W as beta and Y_pred as Y_hat.

Apparently, as far as conventions go both feature vector X and weight vector W are assumed as column vectors. Although this is not important in your case but when we use Neural Nets, this is particularly important, and a weight vector for a layer is given by p*N_n where p is the number of features being input from the previous layer and N_n is the number of nodes in the layer. Check a NN structure and you will understand what I am saying.

As far as the special case in your current question, the book is correct, although comments say Y_predicted should be vector, it is not, since we are doing dot product between X and W (although it is represented in a vectorised form).

Also how can W have dimensions p*K (assuming it is not a NN), it means multiple solutions to the same problem? Instead it is X which will have dimensions p*K and W will have p*1. It means you have K training examples with p features.

If its a NN with K nodes then dimensions for X like p*N and W as p*K is totally possible. The only downside of this notation is that you have to again take a transpose of the result.

Here is the explanation of the notation used:

Notation for NN

$\endgroup$
  • $\begingroup$ So to clarify, is the book misstating the dimensions of W when it says it will be a p*k matrix of coefficients? $\endgroup$ – Hanzy Jul 3 '18 at 3:57
  • $\begingroup$ @Hanzy depends what K is $\endgroup$ – DuttaA Jul 3 '18 at 3:59
  • $\begingroup$ @Hanzy wait let me check the edited question $\endgroup$ – DuttaA Jul 3 '18 at 4:03
  • $\begingroup$ @Hanzy it is correct if K is the number of nodes and incorrect if anything else $\endgroup$ – DuttaA Jul 3 '18 at 4:09
  • 1
    $\begingroup$ K is nodes from above $\endgroup$ – Zafar Kurbonov Jul 3 '18 at 7:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.