0
$\begingroup$

Do we have to use the IOB format on labels in the NER dataset (such as B-PERSON, I-PERSON, etc.) instead of using the usual format (PERSON, ORGANIZATION, etc.)? If so, why? How will it affect the performance of the model?

$\endgroup$
0
$\begingroup$

There's nothing stopping you from training a model with whatever tags you want.

Using what you describe as "usual" format means you would have approx half as many tags as using the IOB format. In theory this means your model will develop higher accuracy faster and with less training data. On the downside, you will need to do more work when interpreting the results in order to be confident where one named entity ends and another one begins.

I made a notebook to do some empirical tests on this which uses 17 tags in the IOB format and 9 in the "usual" format.

TL;DR using the "usual" format did not produce a noticeable improvement in model quality.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.