Although this question is slightly primarily opinion-based and too broad (and I will probably close it as such) and a good answer will necessarily depend on your background, I will list some of the main theoretical prerequisites that everyone should ideally be familiar with before diving into TensorFlow Probability (TFP).
I am familiar with TFP, given that I've been using it for a project, but I've not used all of its functionalities, such as the bijectors. I've only used the Bayesian layers, distributions, etc., so I will try to give an answer based on my experience.
You definitely need to be familiar with the basic concepts of probability theory, such as distributions, random variables, expectations, etc. A full university-level course in probability and statistics would definitely be helpful!
If you want to use Bayesian layers, you also need to be familiar with Bayesian neural networks (BNNs). To understand BNNs, you need to understand the basic concepts of Bayesian inference (BI), given that most BNNs are based on BI. If you are already familiar with variational auto-encoders (VAEs), then it will be easier to understand BNNs. To understand BNNs, you should start reading the paper Weight Uncertainty in Neural Networks. If this paper is difficult to follow, you should first read the paper Variational Inference: A Review for Statisticians and then it may also be useful to read and be familiar with the VAE paper. Therefore, you will need to be familiar with concepts such as Kullback–Leibler divergence and Monte Carlo sampling.
The blog post Regression with Probabilistic Layers in TensorFlow Probability is the first blog post you should read. If you don't understand it, then it probably means you need to learn its prerequisites (i.e. you need to be familiar with at least linear regression).
TFP also provides several implementations of other more advanced concepts such as BNNs. The easiest example to follow is probably the logistic regression example. LR is relatively easy to follow compared to the topics I mentioned above, but, of course, you need to be familiar with logistic regression.
Of course, I cannot list all the theoretical prerequisites (and that's why your post is too broad!) and there are definitely others, but this is a start. Bear in mind that these are not trivial topics, so it is normal to get stuck.