I've found the answer in the original BERT git repo
***** New May 31st, 2019: Whole Word Masking Models *****
This is a release of several new models which were the result of an
improvement the pre-processing code.
In the original pre-processing code, we randomly select WordPiece
tokens to mask. For example:
the man jumped up , put his basket on phil ##am ##mon ' s
head Original Masked Input: [MASK] man [MASK] up , put his [MASK] on
phil [MASK] ##mon ' s head
The new technique is called Whole Word Masking. In this case, we
always mask all of the the tokens corresponding to a word at once. The
overall masking rate remains the same.
Whole Word Masked Input:
the man [MASK] up , put his basket on [MASK] [MASK] [MASK] ' s head
The training is identical -- we still predict each masked WordPiece
token independently. The improvement comes from the fact that the
original prediction task was too 'easy' for words that had been split
into multiple WordPieces.
This can be enabled during data generation by passing the flag