I am making a labeled dataset of images from web streams for a CNN classification. Pictures from the same stream are quite similar as far as background, but slightly different as far as the main object. The focus of what should be learned is in the main object.
My concern is that feeding similar images with the same features in the background will result into making those features more relevant and hence lower the weights of the features that matter.
So, should I be worried about removing similar images from a dataset, so that unrelated features are not learned?
An ideal answer would discuss the trade-offs.
I am aware of the practice of augmenting the training images by scaling/skewing/flipping them around. So it looks like people do it intentionally, but why?
I should also say that it's not about learning from a single stream, there are tons of them. So most images are very square-distant from one another since they are coming from different streams, except those ones that were snapped from the same stream.