Is there a convention on how the input data and the weights are multiplied? The input data can be anything, including the result from the previous layers.

There are two options:

Option 1:

$$\begin{bmatrix}i_1 & i_2\end{bmatrix} \times \begin{bmatrix} w_1 & w_2 & w_3\\w_4 & w_5 & w_6\end{bmatrix} = \begin{bmatrix}i_1*w_1 + i_2*w_4 & i_1*w_2+i_2*w_5 &i_1*w_3+i_2*w_6\end{bmatrix}$$

Option 2:

$$\begin{bmatrix} w_1 & w_4\\ w_2 & w_5\\ w_3 & w_6\end{bmatrix} \times \begin{bmatrix}i_1 \\ i_2\end{bmatrix} = \begin{bmatrix}i_1*w_1 + i_2*w_4 & i_1*w_2+i_2*w_5 &i_1*w_3+i_2*w_6\end{bmatrix}$$


The conventions I have seen tend to post-multiply rather than pre-multiply, although there are examples in the literature which adopt the opposite convention.

Some examples include:

  1. In Deep Learning: An Introduction for Applied Mathematicians, a layer with input $x \in \mathbb R^n$ and output $f(x) \in \mathbb R^m$ is computed by $$ f(x) = \sigma(Wx + b)$$ for a matrix $W \in \mathbb R^{m \times n}$ of weights, a vector $b \in \mathbb R^m$ of biases and an activation function $\sigma \colon \mathbb R^m \to \mathbb R^m$.

  2. In Deep Learning by Goodfellow, Bengio and Courville, they compute a layer in chapter 6 (see p. 171) using

$$ f(x) = \max \{ 0, W^T x + c \},$$

where the $\max$ function is applied componentwise and gives the ReLU activation.

  1. In the paper Federated Learning with Matched Averaging, the authors describe a fully-connected layer using pre-multiplication,

$$f(x) = \sigma(xW),$$

where they omit the bias to simplify notation and as before apply $\sigma$ entrywise. In this case of course we need to view $x$ as a row vector and will obtain a row vector as the result.

Ultimately, as long as you are clear it doesn't matter a great deal: in the end, all that changes is the shape and values of the matrix.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.