Which is more popular/common way of representing a gradient in AI community: as a row or column vector?

Question

Consider the following remark about writing gradients from the chapter named Vector Calculus from the test book titled Mathematics for Machine Learning by Marc Peter Deisenroth et al.

Remark (Gradient as a Row Vector). It is not uncommon in the literature to define the gradient vector as a column vector, following the convention that vectors are generally column vectors. The reason why we define the gradient vector as a row vector is twofold: First, we can consistently generalize the gradient to vector-valued functions $f: \mathbb{R}^n \rightarrow \mathbb{R}^m$ (then the gradient becomes a matrix). Second, we can immediately apply the multi-variate chain rule without paying attention to the dimension of the gradient.

The above remark implicitly says that there is no standard in writing the gradients. So, I can write a gradient of a scalar-valued multivariate function either as a column vector or as a row vector.

But, I want to know which is more common in the AI community?

Answer 1

The issue doesn't come up terribly often. If you are only dealing with vectors, everything is either a row or column vector. It makes no difference which it is.

A more relevant issue is whether one uses the so-called "numerator layout" or "denominator layout". In the numerator layout, $\partial f / \partial x$ is $\mathbb{R}^{n\times m}$, where $f \in \mathbb{R}^n$, $x \in \mathbb{R}^m$. The denominator layout is the other way ($\partial f / \partial x$ is $\mathbb{R}^{m \times n}$).

If you think of the gradient as a row vector, then you are using numerator layout.

I think the numerator layout is overall more common. But there is no real convention, and if the paper content deals with Jacobian matrices it will definitely be mentioned somewhere in the text which layout they use.