0
$\begingroup$
from zlib import crc32

def is_id_in_test_set(identifier, test_ratio):
    return crc32(np.int64(identifier)) < test_ratio * 2**32

def split_data_with_id_hash(data, test_ratio, id_column):
    ids = data[id_column]
    in_test_set = ids.apply(lambda id_: is_id_in_test_set(id_, test_ratio))
    return data.loc[~in_test_set], data.loc[in_test_set]



housing_with_id = housing.reset_index()  # adds an `index` column
train_set, test_set = split_data_with_id_hash(housing_with_id, 0.2, "index")

book says

compute a hash of each instance’s identifier and put that instance in the test set if the hash is lower than or equal to 20% of the maximum hash value. This ensures that the test set will remain consistent across multiple runs, even if you refresh the dataset. The new test set will contain 20% of the new instances, but it will not contain any instance that was previously in the training set.

I don't understand how if the identifier is less than 20% of the maximum hash value will guarantee that an instance does not repeat in the test set

$\endgroup$
5
  • $\begingroup$ Could you clarify what you mean by "repeat"? Note the quoted text doesn't use that word, or make any claims about data repetition. It makes claims about instances with ids. It also doesn't say anything about comparing an id with a hash. $\endgroup$ Jan 30 at 21:45
  • $\begingroup$ @NeilSlater The new test set will contain 20% of the new instances, but it will not contain any instance that was previously in the training set. $\endgroup$ Jan 31 at 15:11
  • $\begingroup$ OK, that's the claim in the the quote from the book. But what do you mean by the word repeat in the phrase "does not repeat in the test set"? What would it mean for an instance to "repeat in the test set"? I am reasonably confident that I understand what the book is trying to say in the quote. I am not confident that I understand what your misunderstanding is, if that makes sense . . . and that's because you are using this word "repeat", and I am not sure what you mean by an instance "repeating". Normally when an item repeats, it means it appears twice or more $\endgroup$ Jan 31 at 15:33
  • $\begingroup$ That makes sense, but as each instance only appears once, and is either in the test set or the training set, it is not surprising, and is not really related to what the book is claiming. $\endgroup$ Jan 31 at 22:03
  • $\begingroup$ sorry an instance can't be on the test set if it was on the test set already $\endgroup$ Feb 1 at 12:29

0

You must log in to answer this question.

Browse other questions tagged .