Was the corruption of Microsoft's "Tay" chatbot an example of catastrophic forgetting?

Question

Tay was a chatbot, who learned from Twitter users.

Microsoft's AI fam from the internet that's got zero chill. The more you talk the smarter Tay gets. — Twitter tagline.

Microsoft trained the AI to have a basic ability to communicate, and taught it a few jokes from hired comedians before setting it lose to learn from its conversations.

This was a mistake.

But why did Tay go so wrong? Was this an example of catastrophic forgetting, where short, recent trends override large, less recent training, or was it something else entirely?

Answer 1

It was essentially a lack of control over crowd-sourced training data.

While Tay was initially set up with some conversational ability, it seemed to be programmed to learn from interactions with other users. Once users became aware of this, they basically gamed the bot by exposing it to inappropriate language, which Tay's algorithms then picked up and repeated. According to the Wikipedia article on the topic, it is not known for sure whether its repeat after me facility was solely at fault, or if there was other behaviour that caused it.

It's not really an example of catastrophic forgetting; for once we don't know how Tay worked internally. I would think it's just that it was overwhelmed by new data coming in which was different from the pre-set. It seems unlikely that the kind of language it was exposed to was in any way known in advance and part of its training set (and labelled as 'inappropriate').

Essentially, the lesson from this is to never trust any unvetted input data for training, unless you want to risk people abusing this trust as happened in this case.

Answer 2

Looking at what happened, it was something similar. Though, the case differs in my eyes from one perspective: if it could only do a few comedy jokes, that probably is not a profound starting point to excel in Twitter.

Firstly, Twitter is about real life, not about comedy. Discussions are sometimes tough and you easily end up to Social Media Bubbles, where only a certain kind of speaking style and topics is cultivated. So, even humans get on the wrong track there; why not a newbie bot? And, with jokes you would probably catch something about language itself, but not about the topics. So, becoming a jerk instead of a nice comedian is a logical direction, where the bot even has to go, at least a little, to communicate on same level, and not alone.

The comedian dataset compared to a Twitter dataset is also very small in a technical sense, so talking about a mini trend overkilling a megatrend in this case is probably not true, because of amounts of examples available.

So, catastrophic things happened in learning, but not catastrophic learning with the definition of that term.