Only the first convolutional layer, with filters that process the input [colour] channels directly, can be rendered directly as image patches in the same domain as the input. The left-most panel in your example looks like that.
Further layers of the neural network cannot be rendered like this for two reasons:
They have a number of input channels based on ...
"Ideally, I would end up with an equation that would allow me to perform the classification without the network".
If you could find such analytic equation without machine learning then why training a multi layer perceptron in the first place? Or to phrase it differently, the mlp you trained is that equation. And I'm not trying to be ironic, if you ...
Yes, due to this issue, you should use temperature scaling after training your model. It will calibrate your probability and you will start to get the same kind of distributions. Here are a good article along with implementation on it.
I think you are misreading the relevant passage here.
Since you do not specify exact excerpt(s), I take that by "implicit assumption" you refer to the equation (2) (application of a ReLU) and the corresponding text explanation (bold emphasis mine):
We apply a ReLU to the linear combination of maps because we are only
interested in the features ...