Introduction: MAP finds a point estimate!
As opposed to your apparently current belief, in maximum a posteriori (MAP) estimation, you are looking for a point estimate (a number or vector) rather than a full probability distribution. The MAP estimation can be seen as a Bayesian version of the maximum likelihood estimation (MLE). Therefore, I will first remind you of the objective of MLE.
Maximum likelihood estimation (MLE)
Let $\theta$ be the parameters you want to find. For example, $\theta$ can be the weights of your neural network. In MLE, we want to find a point estimate (rather than a full distribution). The objective in MLE is
\begin{align}
\theta^*
&= \operatorname{argmax}_\theta p(X \mid \theta) \tag{1}\label{1}
\end{align}
where $p(X \mid \theta)$ is the likelihood of the data $X$ given the parameters $\theta$. In other words, we want to find the parameters $\theta$ such that $p(X \mid \theta)$ is the highest, where $X$ is your given training data, so $X$ is fixed.
The notation $p(X \mid \theta)$ can be confusing because, in a conditional probability distribution, $p(a\mid b)$, we often assume that $b$ is given and $p(a\mid b)$ is a distribution over $a$. However, in the case of MLE, $\theta$ in $p(X \mid \theta)$ is not fixed, but it is a variable, while $X$ is given and fixed. Hence we call $p(X \mid \theta)$ a likelihood rather than a probability density or mass function. Moreover, we often denote the likelihood as $\mathcal{L}(\theta; X) = p_{\theta}(X)$ (and there are other notations, but this is, in my opinion, the least confusing one), because we want to emphasize that the likelihood is actually a function of the variable $\theta$. However, this is notation can also be confusing because we equate a function of a variable $\theta$ to a probability distribution over $X$. However, you should note that $p_{\theta}(X)$ is parametrized by $\theta$.
Therefore, the MLE estimation \ref{1} can also be written as follows
\begin{align}
\theta^*
&=
\operatorname{argmax}_\theta \mathcal{L}(\theta; X) \\
&=\operatorname{argmax}_\theta p_{\theta}(X)
\tag{2}\label{2}
\end{align}
where $\theta^*$ is the point estimate of the objective function.
This notation emphasizes the fact that we want to find $\theta$, such that the probability of the given data $X$ is maximized.
Maximum a posteriori (MAP)
MAP is similar to MLE, but the objective is slightly different. First of all, we assume that $\theta$ is a random variable, so we have an associated probability distribution $p(\theta)$.
Recall that the Bayes' rule is the following
\begin{align}
p(\theta \mid X) = \frac{p(X \mid \theta) p(\theta)}{p(X)}
\tag{3}\label{3}
\end{align}
The objective function in MAP estimation is
\begin{align}
\theta^*
&= \operatorname{argmax}_\theta \frac{p(X \mid \theta) p(\theta)}{p(X)} \\
&= \operatorname{argmax}_\theta p(\theta \mid X) \tag{4}\label{4}
\end{align}
Given that $p(X)$ does not depend on $\theta$, for the purposes of optimization, we can ignore it, so equation \ref{4} becomes
\begin{align}
\theta^*
&= \operatorname{argmax}_\theta p(X \mid \theta) p(\theta) \\
&= \operatorname{argmax}_\theta p(\theta \mid X)
\tag{5}\label{5}
\end{align}
which is the MAP objective.
What is the relationship between MLE and MAP?
In MAP, the objective is \ref{5}, which includes a prior over $\theta$, while, in MLE, equation \ref{1}, there is no such thing.
- Therefore, in MAP, we can assume that the parameters $\theta$ follow a certain distribution, thanks to the usage of $p(\theta)$.
In both MAP and MLE, we want to find a point estimate (which can be a number, if you have just one parameter, or a vector of size $N$, if you have $N$ parameters).
MAP is equivalent to MLE if you use a uniform prior, that is, if $p(\theta)$ is a uniform distribution.
Which distribution fits the data $X$?
In MAP, the human (you, me, etc.) chooses the family of distributions. For example, you can assume that your parameters $\theta$ follow a Gaussian distribution, so $p(\theta)$ will be a Gaussian distribution over the parameters. Why do I say "family"? For example, in the case of a Gaussian distribution, you have two parameters that control the shape of the distribution, namely, the mean and variance. Depending on the concrete values of these two parameters, you will have different Gaussian distributions, so you call all these Gaussian distributions a family.
How do you find $\theta$?
To find $\theta^*$, you can use an optimization method like gradient descent or, in certain cases, you can find a closed-form solution. See also Which distributions have closed-form solutions for maximum likelihood estimation?.
Resources
The following blog post MLE vs MAP: the connection between Maximum Likelihood and Maximum A Posteriori Estimation, by Agustinus Kristiadi (a Ph.D. student in machine learning), might also be useful, so I suggest you read it. It will give you more details that I've left out on purpose to avoid cluttering this answer.
