I'm doing a model to detect duplicate in my database (there is a lot of features that can be different but mean same object in the end)

So I have my feature vector for my duplicate dataset which contains score, distance and relation between 2 identical object that I labelised such as (0, 0, 123, 14000, 5, 10, 0, 0, -1)

Since duplicate are a rare event I was wondering what size of dataset should I use for non duplicate features, since I want my model to learn about the disparity of the multiple features I have I though I should have like 10 times the number of example of non-duplicate and in my model change the weight of duplicate class by multiplying by 10.

Is that a good thing to do or is it better to take 50%/50% of duplicate/non duplicate features for my model ?

Also, should I apply filter to chose my non-duplicate dataset in order to have like close object on some features but different on other. Or should I take them randomly among all the data I have ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.