In supervised machine learning, it is common to say that we learn a function of the form
$$y=g(x) + \epsilon.$$
Generally, $\epsilon$ is used to denote noise or, more precisely, any influence by latent variables such as measurement inaccuracies (right?).
Is it, therefore, correct to say that we use $\epsilon$ to denote the model's imperfection to the real world (caused by anything unknown)?
Yes, precisely.
What do you mention is known in the literature as the Bayes Error. See page top of the page 114 https://www.deeplearningbook.org/contents/ml.html.
The ideal model is an oracle that simply knows the true probability distribution that generates the data. Even such a model will still incur some error on many problems, because there may still be some noise in the distribution. In the case of supervised learning, the mapping from $\mathbf{x}$ to $y$ may be inherently stochastic, or $y$ may be a deterministic function that involves other variables besides those included in $\mathbf{x}$. The error incurred by an oracle making predictions from the true distribution $p(\mathbf{x}, y)$ is called the Bayes error.
Regardless of how clever is your model, the best error you can achieve for the prediction on the data distribution is $\varepsilon$. Note that it holds for the whole data distribution, not a sample of data.
Say, you would like to fit something like $\sin(x) + \varepsilon$. There are 10 points, and one can fit them perfectly with the 9th-degree polynomial, but this an error on training data, and, in case one samples more data points, the error will be likely do exceed the optimal $\varepsilon$.
