You are correct.
The main conceptual difference is that optimization is about finding the set of parameters/weights that maximizes/minimizes some objective function (which can also include a regularization term), while regularization is about limiting the values that your parameters can take during the optimization/learning/training, so optimization with regularisation (especially, with $L_1$ and $L_2$ regularization) can be thought of as constrained optimization, but, in some cases, such as dropout, it can also be thought of as a way of introducing noise in the training process.
You should use regularisation when your neural network is big (where big, of course, is not well-defined) and you have little data (where little is also not well-defined). Why do you want to use regularisation in this case? Because big neural networks have more capacity, so they can memorize the training data more easily (i.e. they can over-fit the training data). If the training data is not representative of the whole data distribution (that you are trying to capture with your neural network), then your neural network may fail on other data from that data distribution, i.e. it may not generalize. Regularization techniques, such as $L_1$ and $L_2$ penalties and dropout, limit the complexity of the functions that can be represented by your neural network (remember that, for a specific set of weights $\theta$, a neural network represents a specific function $f_\theta$), so they prevent your neural network from learning a complicated function that would just over-fit the training data.
Of course, you can also use regularisation when you have a very large dataset or your neural network is not that big. However, in principle, the case mentioned above is the case where you will likely need some kind of regularisation. So, as a rule of thumb, the bigger your NN is and the smaller your training dataset is, the more likely you will need some kind of regularisation.
The typical way to visualize that your neural network is over-fitting is to look at the evolution of the loss function (a type of objective function, which you want to minimize) for the training dataset and the validation dataset (a dataset that you do not use for training, i.e. updating the weights of the neural network). If the training loss is very small, while the validation loss is bigger (stays constant or even increases), as you train more the neural network, that's a good sign of over-fitting, and it suggests that you may need some kind of regularisation. However, note that, even with regularisation, it is not guaranteed that you will find a good set of weights that is able to generalize to all data (not seen during training), but it's more likely.
There are other forms of regularization, such as the KL divergence in the context of Bayesian neural networks or variational auto-encoders, but these are more advanced topics that you don't need to know now. In any case, the KL divergence has the same role as the other regularisation techniques that I mentioned above, i.e., in some way, it restricts/limits the possible functions that you can learn.
