I want to tokenize a text and also want to consider space and new line as separate tokens:

if there are multiple spaces, then tokenize them separately, for example, "Hi, I am\n Teddy" this should tokenize as ["Hi"," ", ","," ","I"," ","am","\n"," "," ","Teddy"]

Note: That After new-line (\n) in text there are 2 spaces


1 Answer 1

def tokenize_text(text):
    return re.findall(r'\S+|\n|\s', text.replace('\n\n', '\n \n'))

This gives the tokenization of text, including space and new-line


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .