You can know it better, if you know the concept of entropy:
Information entropy is the average rate at which information is produced by a stochastic source of data. The information content (also called the surprisal) of an event ${\displaystyle E}$ is an increasing function of the reciprocal of the ${\displaystyle p(E)}$ of the event, precisely ${\displaystyle I(E)=-\log _{2}(p(E))=\log _{2}(1/p(E))}$. Shannon defined the entropy Η of a discrete random variable ${\textstyle X}$ with possible values ${\textstyle \left\{x_{1},\ldots ,x_{n}\right\}}$ and probability mass function ${\textstyle \mathrm {P} (X)}$ as:
$${\displaystyle \mathrm {H} (X)=\operatorname {E} [\operatorname {I} (X)]=\operatorname {E} [-\log(\mathrm {P} (X))].}$$
Here ${\displaystyle \operatorname {E} }$ is the expected value operator, and $I$ is the information content of ${\displaystyle I(X)}$ is itself a random variable. The entropy can explicitly be written as
$${\displaystyle \mathrm {H} (X)=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{b}\mathrm {P} (x_{i})}}$$
where $b$ is the base of the logarithm used.
Now, the KL divergence, try to find the cross entropy of two probability distributions.
So, what is the cross entropy:
The cross entropy between two probability distributions ${\displaystyle p}$ and ${\displaystyle q}$ over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution ${\displaystyle q}$, rather than the true distribution ${\displaystyle p}$.
The cross entropy for the distributions ${\displaystyle p}$ and ${\displaystyle q}$ over a given set is defined as follows:
$${\displaystyle H(p,q)=\operatorname {E} _{p}[-\log q]}.$$
The definition may be formulated using the Kullback–Leibler divergence${\displaystyle D_{\mathrm {KL} }(p\|q)}$ of ${\displaystyle q}$ from ${\displaystyle p}$ (also known as the relative entropy of ${\displaystyle p}$ with respect to ${\displaystyle q}$).
$${\displaystyle H(p,q)=H(p)+D_{\mathrm {KL} }(p\|q)},$$
where ${\displaystyle H(p)}$ is the entropy of ${\displaystyle p}$.
Now, as we want to approximate a distribution function $p$ with other distribution function $q$, we want to minimize the cross entropy between these two. The first part $H(p)$ could not be changed as we want to find a distribution $q$ and $p$ is given. Hence, we need to minimize $KL$ divergence of these two, to minimize the cross entropy and better approximation for the $p$ distribution.
