Yes, you are right that you just zero-pad to get the right dimensions. The operation in color space is just a scalar product, so that you could get the backwards operation in that dimension also just per the formulas of the matrix-vector product.
Note that the convolution operations forward and backwards are different.
The forward pass of the convolution layer has two elementary steps: first the convolution operation and then the cutting out of the fully convolved center sequence.
\begin{align}
z&=c*_{rev}x\\
y&=P_Nz
\end{align}
Let's just consider the one-dimensional case, higher dimensions proceed similarly. Then $P_N$ cuts out the finite sequence with support $N$, $y_n=z_n$ if $n\in N$, else $y_n=0$. Similarly for the projections to the support $K$ of the coefficient sequence $c$ and $M$ of the input sequence $x$.
The convention for CNN is to not use the convolution as used for polynomial multiplication, where $[a*b]_n=\sum_ka_kb_{n-k}$, but to have it look more like a scalar product,
$$
[c*_{rev}x]_n = \sum_{m\in M} c_{m-n}x_m=\sum_{k\in K}c_kx_{k+n}=[rev(c)*x]_n,
$$
$rev$ for "reverse", where $[rev(c)]_k=c_{-k}$.
Note that this version of the convolution product is not commutative.
These all are linear operations, so in principle the gradient back propagation has the same structure as the usual matrix multiplication.
The structural principle of gradient computation is that gradients $\bar u^T=\bar L\frac{\partial L}{\partial u}$ act as linear functionals on tangent vectors $\dot u$, and that the scalar value $\langle \bar u,\dot u\rangle$ of it is independent on where in the graph the pairing of gradient and tangent is carried out,
$$
⟨\bar y,\dot y⟩=⟨\bar z,\dot z⟩=⟨\bar x,\dot x⟩+⟨\bar c,\dot c⟩.
$$
The tangent propagation is, using the product rule,
\begin{align}
\dot z&=\dot c*_{rev}x+c*_{rev}\dot x\\
\dot y&=P\dot z
\end{align}
Evaluating the defining relations gives
\begin{align}
⟨\bar z,\dot z⟩&=⟨\bar y,\dot y⟩=⟨\bar y ,P\dot z⟩=⟨P_N^T\bar y,\dot z⟩ \\
⟨\bar x,\dot x⟩+⟨\bar c,\dot c⟩&=⟨\bar z,\dot z⟩=⟨\bar z,(\dot c*_{rev}x)⟩+⟨\bar z,(c*_{rev}\dot x)⟩
\end{align}
The first operation $\bar z=P^T\bar y$ is just zero-padding of the sequence as $(0_{k_1},\, y^T,\, 0_{k_2})^T$. If one thinks of all sequences continued indefinitely by zero, then $P_N^T$ does nothing, as the zeros for the padding are already present.
To analyze the combination of scalar product and convolution, let's go into the details
\begin{align}
⟨\bar z,(\dot c*_{rev}x)⟩
&=\sum_{n\in N}\bar z_n\sum_{k\in K}\dot c_kx_{n+k} \\
&=\sum_{k\in K}\dot c_k\sum_{n\in N}\bar z_nx_{k+n} \\
&=⟨(\bar z*_{rev}x),\dot c⟩ \\
\implies \bar c &= P_K(\bar z*_{rev}x)
\end{align}
as $\dot c$ always only has support $K$, so the support of the gradient to it needs to have the same support.
For the second term one gets similarly
\begin{align}
⟨\bar z,(\dot c*_{rev}x)⟩
&=\sum_{n\in N}\bar z_n\sum_{m\in M}c_{m-n}\dot x_m \\
&=\sum_{m\in M}\dot x_m\sum_{n\in N}\bar z_nc_{m-n} \\
&=⟨(\bar z*c),\dot x⟩ \\
\implies \bar x &= P_M(\bar z*c)
\end{align}
