Is there a convention on the order of multiplication of the weights with the inputs in neural nets?

Question

Is there a convention on how the input data and the weights are multiplied? The input data can be anything, including the result from the previous layers.

There are two options:

Option 1:

$$\begin{bmatrix}i_1 & i_2\end{bmatrix} \times \begin{bmatrix} w_1 & w_2 & w_3\\w_4 & w_5 & w_6\end{bmatrix} = \begin{bmatrix}i_1*w_1 + i_2*w_4 & i_1*w_2+i_2*w_5 &i_1*w_3+i_2*w_6\end{bmatrix}$$

Option 2:

$$\begin{bmatrix} w_1 & w_4\\ w_2 & w_5\\ w_3 & w_6\end{bmatrix} \times \begin{bmatrix}i_1 \\ i_2\end{bmatrix} = \begin{bmatrix}i_1*w_1 + i_2*w_4 & i_1*w_2+i_2*w_5 &i_1*w_3+i_2*w_6\end{bmatrix}$$

Answer 1

The conventions I have seen tend to post-multiply rather than pre-multiply, although there are examples in the literature which adopt the opposite convention.

Some examples include:

1. In Deep Learning: An Introduction for Applied Mathematicians, a layer with input $x \in \mathbb R^n$ and output $f(x) \in \mathbb R^m$ is computed by $$ f(x) = \sigma(Wx + b)$$ for a matrix $W \in \mathbb R^{m \times n}$ of weights, a vector $b \in \mathbb R^m$ of biases and an activation function $\sigma \colon \mathbb R^m \to \mathbb R^m$.

2. In Deep Learning by Goodfellow, Bengio and Courville, they compute a layer in chapter 6 (see p. 171) using

$$ f(x) = \max \{ 0, W^T x + c \},$$

where the $\max$ function is applied componentwise and gives the ReLU activation.

3. In the paper Federated Learning with Matched Averaging, the authors describe a fully-connected layer using pre-multiplication,

$$f(x) = \sigma(xW),$$

where they omit the bias to simplify notation and as before apply $\sigma$ entrywise. In this case of course we need to view $x$ as a row vector and will obtain a row vector as the result.

Ultimately, as long as you are clear it doesn't matter a great deal: in the end, all that changes is the shape and values of the matrix.