Why is it called back-propagation?
I don't think there is anything special here!
It's called back-propagation (BP) because, after the forward pass, you compute the partial derivative of the loss function with respect to the parameters of the network, which, in the usual diagrams of a neural network, are placed before the output of the network (i.e. to the left of the output if the output of the network is on the right, or to the right if the output of the network is on the left).
It's also called BP because it is just the application of the chain rule. Why is this interesting?
Let me answer this question with an example. Consider the function $y=e^{\sin(x^{2})}$. This is a composite function, i.e. a function composed of multiple simpler functions, which, in this case, are $e^x$, $\sin(x)$, $x^2$ and $x$. To compute the derivative of $y$ with respect to $x$, let's define the following variables
\begin{align}
y &= f(u) = e^u,\\
u &= g(v) = \sin v = \sin(x^2),\\
v &= h(x) = x^2
\end{align}
The derivative of $y$ with respect to the variable $x$ is (according to the chain rule)
$$
\underset{\color{red}{\LARGE \rightarrow}}{
\frac{dy}{dx} = \frac{dy}{du} \color{green}{\cdot} \frac{du}{dv} \color{green}{\cdot}
\frac{dv}{dx}}
$$
If you read this equation from the left to the right, you can see that we are going backward (i.e. from the function $y$ to the function $v$). This is the same thing with BP!
Why is it called "chain rule"? Because you are chaining different partial derivatives. More specifically, you are multiplying them.
BP is also known as the reverse mode of automatic differentiation. Why? The automatic differentiation should be self-explanatory, given that the BP algorithm is just the computation of partial derivatives, and you do this automatically, i.e. with a program, rather than by hand. The expression "reverse mode" refers to the fact that we compute the derivatives from the outer function (which, in the example above, is $e^x$) to the inner function (which, in the example above, is $x$). The Wikipedia article related to automatic differentiation provides more details.
What exactly are you back-propagating?
The partial derivative of the loss function $\mathcal{L}$ with respect to a parameter $w_i$, i.e. $\frac{\partial \mathcal{L}}{\partial w_i}$, intuitively, represents the "contribution" of the parameter $w_i$ to the loss. After having computed these partial derivatives (i.e. the gradient), you use gradient descent to update each parameter $w_i$ as follows
$$
w_i \leftarrow w_i - \gamma \frac{\partial \mathcal{L}}{\partial w_i}
$$
where $\frac{\partial \mathcal{L}}{\partial w_i}$ represents WHAT we propagatED, which is the error (or loss) that the neural network makes.
This gradient descent step will hopefully make your network produce a smaller error next time.
The modern version of back-propagation was published (in 1970) by a Finnish master's student called Seppo Linnainmaa, but he didn't reference neural networks. This Jürgen Schmidhuber's article goes into the details of the history of BP.
