Gradient descent (GD) is an optimisation algorithm, that is, it is used to find a (local) minimum of a multi-variable and differentiable function $f$. GD is an iterative and numerical optimisation algorithm. It is iterative because it proceeds in iterations. It is numerical because it is not an algorithm which produces an exact solution, due to numerical errors (e.g., round-off errors).
GD is based on the idea that, if a (multi-variable) function $f(\mathbf{x})$ (where $\mathbf{x} \in \mathbb{R}^N$ is a variable representing the input of $f$) is defined and differentiable in a "neighbourhood" of a point $\mathbf{\theta} \in \mathbb{R}^N$, which is a point (in the loose sense of the word "point") in the domain of $f$ (that is, $f(\mathbf{\theta})$ is defined and differentiable), then $f$ decreases fastest if one goes from $\mathbf{\theta}$ in the "direction" of the negative gradient of $f$ at $\mathbf{\theta}$, that is, $-\nabla f(\mathbf {\theta})$. (Do not be scared of the symbol $\theta$, we could have denoted it by $\mathbf{a}$!). Let us break this dense definition down!
Note that here we assume that $f$ is a multi-variable function (that is, a function whose input is not just one real number, but it is a function of $N$ real numbers, which can be grouped in a vector, e.g. $\mathbf{\theta} \in \mathbb{R}^N$), hence $\mathbf{x}$ and $\mathbf{\theta}$ are vectors of real numbers. To emphasize, $\mathbf{\theta}$ is some specific (and "fixed") point in the domain of $f$, whereas $\mathbf{x}$ is a variable that can take any of these specific points.
The basic statement/assignment which is iterated in GD, which captures the definition above, is the following:
$$\mathbf{\theta}_{n+1} \gets \mathbf{\theta}_{n} - \gamma \nabla f(\mathbf{\theta} _{n}) \label{gd-assignment}\tag{1}$$
where the $\gamma$ is a hyper-parameter which represents the "strength" of the step we take in the direction of the negative gradient (i.e., $-\nabla f(\mathbf{\theta} _{n})$). Note the "minus" (i.e., $-$) in front of $\nabla f(\mathbf{\theta}_{n})$, hence the expression "in the direction of the negative gradient". Note also that here "direction" is implemented as a subtraction. The subscript $_n$ represents the current iteration number.
As simple as it looks, the GD algorithm consists in the execution of the assignment \ref{gd-assignment} until some condition is met (e.g., ${\theta}_{n+1}$ is "close enough" to ${\theta}_{n}$). In a nutshell, GD is an iterative algorithm which updates $\theta$ by iterating \ref{gd-assignment} multiple times.
In the context of machine learning (ML), in particular supervised learning, we initially have an untrained model $\mathcal{M}$ (e.g., a neural network), that is, a model whose parameters (sometimes also called "weights") are e.g. randomly initialised. To train $\mathcal{M}$ (i.e., to find the most appropriate parameters of the model with respect to the training data), we first define a so-called "loss function" (which is differentiable), denoted by $\mathcal{L}$, which represents the discrepancy between the current output of $\mathcal{M}$ and the expected output (with respect to the training data) of $\mathcal{M}$. The process of training a model consists in minimising this loss function, that is, it consists in finding the minimum of $\mathcal{L}$.
To do this, as we have just seen above, we can use GD. If you consider the learning process the one of minimising this loss function $\mathcal{L}$, then any algorithm which minimises $\mathcal{L}$ should be called a "learning algorithm". Hence GD should be called a "learning algorithm".
Note that the parameters of the model $\mathcal{M}$, which need to be found during the training process, are often denoted, in ML, by $\theta$. So, the reason why I used the strange symbol $\theta$ in the assignment $\ref{gd-assignment}$ should now be clearer.
Back-propagation (BP) is just an algorithm, proposed by Seppo Linnainmaa in his master's thesis, to compute the derivative of a differentiable (composite) function, which can be represented as a graph.
In ML, back-propagation is often used to compute $\nabla f(\mathbf{\theta}_{n})$ in the assignment \ref{gd-assignment}. So, BP is used as a sub-routine in the GD algorithm (or any other optimization algorithm) to find the parameters of the model $\mathcal{M}$.
Note that, in the context of ML, $\gamma$ in \ref{gd-assignment} is often called the "learning rate". There's a reason for this: GD is the actual learning algorithm. BP is just a sub-routine used to compute the gradient, but we could have used another sub-routine. We could say, in the context of ML (and, in particular, NNs), that the combination of GD and BP is the actual "learning algorithm", but, if by "learning" we mean "optimisation", then the actual "learning algorithm" is GD and not BP, because BP does
not optimise anything alone.
