# Why do language models place less importance on punctuation?

I have very outdated idea about how NLP tasks are carried out by normal RNN's, LSTM's/GRU's, word2vec, etc to basically generate some hidden form of the sentence understood by the machine.

One of the things I have noticed is that in general researchers are interested in generating the context of the sentence, but oftentimes ignore punctuation marks which is on of the most important aspects for generating context. For example:

“Most of the time, travellers worry about their luggage.”

“Most of the time travellers worry about their luggage”

Source

Like this there exists probably 4 important punctuation marks .,? and !. Yet, I have not seen any significant tutorials/blogs on them. It is also interesting to note that punctuations don't have a meaning (quite important, since most language models try to map word to a numerical value/meaning), they are more of a 'delimiter'. So what is the current theory or perspective on this? And why is it ignored?

Language models almost always map every word to an embedding. There are many embedding algorithms with most of them having interpolation properties i.e. If $$E(word)$$ represents embedding of a word then $$E(king)-E(male)+E(female) \sim E(queen)$$. The smoother the interpolation properties the better the model understands the word, these properties don't exactly make much sense when it comes to delimiters.

Yet, there are instances where a delimiter embedding is learned ( always has an embedding ). While using these first all punctuation in the text is converted to one specific word, say 'dlmt', the embedding algorithm learns an embedding for this word treating it as if it would have any word. This maintains interpolation properties where the delimiter is understood to be a word that is used to split context.

I have observed that delimiters such as question mark or exclamation marks at the end of sentence are also understood to be breaks in context, in these cases the model learns if the statement is a question or so just by the context given by the words and stops in the sentence

• Different delimiters convey different meaning. So why group them under same category? – DuttaA Jul 26 '19 at 11:27
• The model learns to treat them as stops anyway. Different delimiters works better for sentence embeddings. The most distinct delimiter is '?' which also happens to learn a pretty similar embedding to '.' , '!' or ','; the model cannot associate meaning as such to delimiters much beyond the fact that its a break word --- This might be due to under-represented examples too – ashenoy Jul 26 '19 at 11:33

You are right. Approaches that map words to meaning solely do fail in this regard. None the less Word2Vec and Glove have shown wonderful downstream results. This in itself may indicate that most of the time, punctuation's addition can be interpolated. But as you provided, there are cases where this just is not true!

Now of days I would say most models actually use almost NO preprocessing. This is surprising but its due to the rise in power of learnable, reversable, tokenizations. Some examples of these include byte pair encoding (bpe) and the sentence piece model (spm).

State-of-the-art NLP generally rely on these. Examples include BERT and GPT2, which are general purpose pretrained Language Models. Their ability to parse and understand (i use this word loosely) a wide variety of phrasing, spelling and more can be partially due to the freedom in the preprocessing.

Takeaway: You can achieve good results by using preprocessing in a manner that will eliminate information but keep the meat and bones that you are interested in (but this requires domain knowledge paired with optimization experience), but the field seems gearing towards models that are more inclusive, more transferable, and dont have the problems you mention by design.

This is a somewhat provocative view, so be warned (and please don't down-vote this if you feel provoked by it!):

In the "old days", when information retrieval (IR) was one of the main tasks in NLP, several categories of words were ignored as stopwords; conjunctions, determiners, prepositions, etc. These function words do not carry meaning themselves, but organise the structure of sentences. Most IR algorithms worked on frequencies of individual words, and as functions words are very frequent (of and the are the two most frequent English words) and don't mean anything by themselves, they were ignored. This kept the index files small and didn't seem to influence the results.

However, if you want to analyse sentences themselves, they are rather important. They are also useful for all sorts of other tasks where you are looking at sequences of words (eg part-of-speech tagging based on context). Similar for word embeddings: without function words you'd not have any meaningful context to work with. So, increasingly you would not ignore function words anymore.

My suspicion is, that punctuation is now in the 'stopword position': it's not too clear how it influences meaning, and is often inconsistent or redundant (obviously not in all cases). So you can probably treat it as 'noise' and get away with it for most applications. For example, looking at meanings of words, it probably doesn't matter that much whether the sentence they occurred in was a question or an exclamation. By removing punctuation (maybe apart from sentence-terminators), your model is a bit smaller and you don't lose much.

Since punctuation is purely a property of written language, we can generally get away without it, as we do in speech. A text without punctuation might be harder to read, because we're not used to it, but don't forget that some writing systems (Chinese, Egyptian hieroglyphics, ...) don't even have spaces between words — and people can still use them without problems.

• I actually like the answer since it takes a statistical viewpoint that in most cases punctuation aren't important. I also suspect that punctuations are more to provide an emotional context rather than for actual meanings, and afaik most language models do not show much emotion but rather hard facts – DuttaA Aug 2 '19 at 16:48