0
$\begingroup$

I am exploring a potential NLP project. I was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the # removed and the word tokenised? Is it tokenised with the #?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Word including Hash tag need to be tokanised. It has a special meaning and given a context. The word by itself has a different meaning.

Improve this answer
$\endgroup$
1
  • $\begingroup$ So have they been encoded during training by models such as GPT or BERT? $\endgroup$
    – user1234
    Commented Feb 18, 2021 at 13:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .