I've recently started reading a book about deep learning. The book is titled "Grokking Deep Learning" (by Andrew W Trask). In chapter 3 (pages 44 and 45), it talks about multiplying vectors using dot product and element-wise multiplication. For instance, taking 3 scalar inputs (vector) and 3 vector weights (matrix) and multiplying.

From my understanding, when multiplying vectors the size needs to be identical. The concept I have a hard time understanding is multiplying vectors by a matrix. The book gives an example of an 1x4 vector being multiplied by 4x3 matrix. The output is an 1x3 vector. I'm am confused because I assumed multiplying vector by matrix needs the same number of columns, but I have read that the matrices need rows equal to the vectors columns.

If I do not have an equal number of columns, how does my deep learning algorithm multiply each input in my vector by a corresponding weight?

  • $\begingroup$ I the deep learning algorithm; I might have vector of three(columns) inputs. I also would have three weights. Each weight would have a vector of three columns to correspond With each input. These weights would be stored in matrix. The box gives an example of element wise multiplication with a 1x4 times 2x5 and states they don’t have same number of columns so it throws error. On the next page it give example of 1x4 times 4x3 and states the output is a 1x3 vector. This is contradicting. $\endgroup$ – z Eyeland Feb 9 '19 at 12:06

In general, when people do not explicitly state it, a vector $v \in \mathbb{R}^n$ is usually considered a "column vector", that is, you can think of it as the matrix $v \in \mathbb{R}^{n \times 1}$ (that is, a matrix with $n$ rows and $1$ column). If it is not explicitly stated and you assume that the given vector is a column vector, but the dimensions do not match, then you should check that the dimensions match if you consider the given vector as a "row vector" (because it might be the case that the author is implicitly considering the vectors as row vectors).

Having said that, you can multiply a vector $v \in \mathbb{R}^n$ by a matrix $A \in \mathbb{R}^{n \times m}$ from the left, that is, you can do $v^T A$. Here, I considered $v$ has a column vector (that is, $v \in \mathbb{R}^{n \times 1}$), even though I have not explicitly stated it: you can and you often need to deduce this from the context! If I transpose $v$, I obtain $v^T \in \mathbb{R}^{1 \times n}$, and, thus, you can see that you can indeed perform the operation $v^T A = u \in \mathbb{R}^{1 \times m}$. Note that, at this point, $u$, the vector resulting from the operation $v^T A$, is actually considered a matrix, but you can still use it as a vector, if you need and that is permitted according to the mathematical rules of the operations you need to perform. Note also that I cannot multiply $v$ from the right of $A$, because there is no way of making the dimensions match. Have a look at this question, if you do not know how to multiply a vector by a matrix from the left.

Similarly, you can multiply $v \in \mathbb{R}^n$ by the matrix $B \in \mathbb{R}^{m \times n}$ only from the right. If $v$ is a column vector (that is, $v \in \mathbb{R}^{n \times 1}$), you need to do $B v \in \mathbb{R}^{m \times 1}$, but, if $v$ is a row vector (that is, $v \in \mathbb{R}^{1 \times n}$), you will first need to transpose it, so that you can perform the operation: $B v^T\in \mathbb{R}^{m \times 1}$.

Furthermore, note that, if you multiply a vector by a matrix from the left, that vector needs to be a "row vector", so, if you initially assume that the vector is a column vector (or that is explicitly stated), you will need to transpose it first, before the multiplication. However, if the vector is already a row vector, you won't have to transpose it. Similarly, if you multiply a vector by a matrix from the right, you will need a column vector.

To conclude, you can multiply a vector either from the left or right of a matrix, but you need to make sure that the dimensions match: if you multiply from the left, you will need to check that the dimensions of the vector match the number of the rows of the matrix; if you multiply the vector from the right of the matrix, you will need to check that the dimensions of the vector match the number of columns of the matrix.

  • $\begingroup$ Not to be harsh, but your answer makes no effort to answer OP's question. Somehow I believe you didn't read the question properly (aside from the question being highly unclear). (On a side note, I would like to say it was actually your comments which taught me to read a question properly instead of skimming). $\endgroup$ – user9947 Feb 9 '19 at 13:33
  • 1
    $\begingroup$ @DuttaA I read the question. I tried to give an answer which is as general and thus, hopefully, useful as possible. In my answer, I explain all the details that the OP is confused about. $\endgroup$ – nbro Feb 9 '19 at 13:35
  • $\begingroup$ in the last 2 paragraphs OP gives conflicting statements that he understands row=columns (required for *)and then says matrix * requires equal columns...And then at the end says inputs will not get multiplied by weights....So I think OP is trying to figure how exactly does the matrix * take care of the fact that weights are * to inputs. $\endgroup$ – user9947 Feb 9 '19 at 13:37
  • $\begingroup$ @DuttaA I will wait for the feedback of the OP. If he's not happy with my answer, I will modify it. If he is also confused about how to structure his weight matrix, then this is another question, which either can be asked separately, or maybe I could also answer it here. $\endgroup$ – nbro Feb 9 '19 at 13:40
  • $\begingroup$ @DuttaA Also note that the OP stated "The concept I’m have hard time understanding is multiplying vectors by a matrix". In my answer, I give him the general guidelines/rules to follow. I also address his statement " Im am confused because I assumed multiplying vector by matrix needs the same number of columns as well but have read that the matrixes need rows equal to the vectors columns.", so the confusion lies in how to multiply vectors by matrices: this is what I supposed from his/her statements. $\endgroup$ – nbro Feb 9 '19 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.