The term you're looking for is "replacement schemes". As far as I'm aware, the primary reference on this is still Replacement Schemes for Transposition Tables, although it is a fairly old paper from 1994.
I'll very briefly summarize the seven different schemes listed in this paper, but full text of the paper is also freely available and contains more info:
- Deep: preserve position for which the deepest subtree below it was searched.
- New: Always replace an old entry with the newest entry.
- Old: Opposite of New. Paper mentions only including it for the sake of completeness, implies that it's probably not a good scheme.
- Big1: similar to Deep, but uses the size (in number of nodes) of subtree searched, rather than just its depth. If the same position (i.e. table entry) occurs multiple times in a subtree, it is only counted once.
- BigAll: same as above, but really counts the number of nodes rather than number of positions (so a position that occurs multiple times is counted multiple times).
- TwoDeep: In this scheme, the TT is modified to contain two positions per table entry, rather than just one position. You could view it as the table having two "layers". In this scheme, a new position is moved into the first "slot" if it has the deepest subtree (as in the Deep scheme), moving the previous position in the first slot into the second slot. If a new position doesn't have a new deepest subtree, it is instead moved into the second slot.
- TwoBig1: Similar to the above, but using Big1-style replacement rather than Deep-style replacement.
If I recall correctly, the Two-layered schemes tend to perform best (of course they do require approximately twice as much memory for the same number of bits per key).