Outside of using a generated dataset to study machine learning, the typical purpose of a trained machine learning model is to process new inputs from some source.
For a model to be effective, the training data set inputs and new inputs should be taken from the same distribution. The loss function used in training, combined with cross-validation to measure and maintain generalisation, will have ensured that the most accurate results occur for a population of inputs that is similar to the training data.
The further that inputs stray from being like the training data, in any aspect, then the more likely that outputs from a trained model are inaccurate. This may include over-generalising - if you set much higher ranges for some variations - e.g. far more dilate and erode that would be seen in practice, then the neural network weights will be tuned to allow for this data and may score worse on your target data even though it will appear to score well overall on the training data. That is because the measurements for loss and accuracy from the more realistic generation will be diluted by measurements from training data that has no relevance to the real-world problem you are trying to solve. Maybe it will be OK, maybe worse, maybe even better - however your measurements of loss and accuracy during training will not tell you.
So there is significant danger in relying on a generated-only dataset for training. If any aspect of the simulated inputs does not match how the system will be used in practice, the impact is likely to be felt in terms of reduced accuracy.
For your digits example, you should consider where the "real" digits will come from later, and try to ensure that your data generation takes into account any complications, variations, imperfections that will occur when collecting the data. For instance, if the real digits are scanned from paper, then take a look at some typical scanned images, and check how close your generated data is to them.
If you can obtain a limited number of "real" values, perhaps not enough for training, but enough to get some accuracy statistics from, then consider using them for test and cross-validation phases. Remember, that using them for test should be done sparingly, and not used to select between models with similar results, but only to establish a rough estimate of accuracy at the end of training. Whilst using any for cross-validation may help select a model that generalises the best between the generated data set and reality, but precludes using the same examples for test.