I'm working on a problem in ML to assess the performance of multiple vendors. I have a set of features in my dataset, and it appears each vendor is characterized by its own distribution. This is my hypothesis, as I see my target variable shifts and changes value ranges with time, as different vendors' data is achieved.
Is there a way, in a real application system, to implement a mechanism to detect such data drift/change (whether in target variable or features distribution)?
If so, how can I deal with it? Should I be constantly re-training a model to deal with the new data? What is the common practices to deal with new observed data?