# Is there a convention on the order of multiplication of the weights with the inputs in neural nets?

Is there a convention on how the input data and the weights are multiplied? The input data can be anything, including the result from the previous layers.

There are two options:

Option 1:

$$\begin{bmatrix}i_1 & i_2\end{bmatrix} \times \begin{bmatrix} w_1 & w_2 & w_3\\w_4 & w_5 & w_6\end{bmatrix} = \begin{bmatrix}i_1*w_1 + i_2*w_4 & i_1*w_2+i_2*w_5 &i_1*w_3+i_2*w_6\end{bmatrix}$$

Option 2:

$$\begin{bmatrix} w_1 & w_4\\ w_2 & w_5\\ w_3 & w_6\end{bmatrix} \times \begin{bmatrix}i_1 \\ i_2\end{bmatrix} = \begin{bmatrix}i_1*w_1 + i_2*w_4 & i_1*w_2+i_2*w_5 &i_1*w_3+i_2*w_6\end{bmatrix}$$

The conventions I have seen tend to post-multiply rather than pre-multiply, although there are examples in the literature which adopt the opposite convention.

Some examples include:

1. In Deep Learning: An Introduction for Applied Mathematicians, a layer with input $$x \in \mathbb R^n$$ and output $$f(x) \in \mathbb R^m$$ is computed by $$f(x) = \sigma(Wx + b)$$ for a matrix $$W \in \mathbb R^{m \times n}$$ of weights, a vector $$b \in \mathbb R^m$$ of biases and an activation function $$\sigma \colon \mathbb R^m \to \mathbb R^m$$.

2. In Deep Learning by Goodfellow, Bengio and Courville, they compute a layer in chapter 6 (see p. 171) using

$$f(x) = \max \{ 0, W^T x + c \},$$

where the $$\max$$ function is applied componentwise and gives the ReLU activation.

1. In the paper Federated Learning with Matched Averaging, the authors describe a fully-connected layer using pre-multiplication,

$$f(x) = \sigma(xW),$$

where they omit the bias to simplify notation and as before apply $$\sigma$$ entrywise. In this case of course we need to view $$x$$ as a row vector and will obtain a row vector as the result.

Ultimately, as long as you are clear it doesn't matter a great deal: in the end, all that changes is the shape and values of the matrix.