The reason is that when using a Convolutionalconvolutional layer, you select the size of the filter kernels, which are independent of the image/layer input size (provided that images smaller than the kernels are padded appropriately).
When using a Densedense layer, you specify the size of the layer itself and the resulting weight matrix is a function of both the size of the Densedense layer and the upstream layer. This is because each neuron in the upstream layer makes a connection to each neuron in the Densedense layer. So, if you have 50 neurons in the upstream layer and 20 neurons in the Densedense layer, then the weight matrix has 50x20=1000$50 \times 20=1000$ values. Those weights are what get determined during the training phase, and so those layer sizes are fixed.
Now, the output of a CNN layer is a number of images/tensors (specified by the number of filters chosen), whose size is determined by the kernel size and any padding option chosen. If those feedare fed into a Densedense layer, then that fixes the the size that those images can be (because of the reason given in the previous paragraph).
On the other hand, if no Densedense layer is used in the whole network, then the input to the first CNN layer can be any size because the weights are just the individual parameters of the filter kernels, and the filter kernels remain the same size regardless of the input tensor size.