10
$\begingroup$

My question relates to full-text translators that are not specifically based on LLMs. My current understanding is that the term Generative AI goes beyond LLMs and that the full-text translators (especially those which are based on artificial neural networks) also fall into this category.

In the Wikipedia article about Generative AI, I could only find the statement that generative AI systems such as ChatGPT are also used for translations. That is quite obvious, but this does not answer the question of whether other full text translators are also commonly referred to as Generative AI.

IBM research defines Generative AI as

Generative AI refers to deep-learning models that can generate high-quality text, images, and other content based on the data they were trained on.

AFAIK DeepL, Google Translate, Bing Translate but also a few not so well-known systems falls clearly into that definition.

My question was triggered by an answer on Meta Stack Overflow that contained the following sentence:

Besides, the banner clearly states: "Answers generated by artificial intelligence tools". Translations aren't generated answers. They're translations.

with the implicit conclusion that it is clear to everyone that pure translators do not count as generative AI, hence it does not need any further explanation. Other participants in that thread seem to agree to that point of view. However, I think their use of terminology is not the typical use in the field of AI, and I would like to hear what the experts say.

$\endgroup$
2
  • 1
    $\begingroup$ I found a duplicate on GenAI SE that answers this. $\endgroup$
    – Cerbrus
    Jan 25 at 10:44
  • $\begingroup$ I found this interesting link about the history of Generative AI. Translators apart from LLMs are not explicitly mentioned. Still it gives a good outline about the >70 years old history of the field, the breakthroughs since 2010, and the wide range of tools for which the term GenAI is used. $\endgroup$
    – Doc Brown
    Jan 25 at 17:21

5 Answers 5

8
$\begingroup$

Generative AI, as defined by IBM research, refers to deep-learning models capable of creating new content, be it text, images, or other media, based on their training data. This definition indeed encompasses models like GPT-3 or GPT-4, which can generate text in various styles and formats, including translations.

However, when it comes to full-text translators like Google Translate, DeepL, or Bing Translate, there's a nuanced difference. These systems are typically based on neural machine translation (NMT) models, a specific application of deep learning tailored for the task of translating text from one language to another. While these NMT systems are indeed 'generative' in the sense that they produce new text in a target language, their primary function is not to create original content but to convert existing content from one language to another as accurately as possible.

$\endgroup$
13
  • 5
    $\begingroup$ I would also say that typical models that people call "Generative AI" like ChatGPT and Dall-E, are not meant to create original content either, so that would not be a good criteria to define generative models. $\endgroup$
    – Dr. Snoopy
    Jan 25 at 13:33
  • 5
    $\begingroup$ Wait... What are they meant to do then, if not "create original content"? $\endgroup$
    – Cerbrus
    Jan 25 at 13:34
  • 4
    $\begingroup$ Depending on the model, they are trained to predict next word given some context (LLMs), or to imitate a training set of natural images (Stable Difussion and Dall-E) , creating new content is a strong claim that is generally unproven. For example see the NYT lawsuit against OpenAI. $\endgroup$
    – Dr. Snoopy
    Jan 25 at 16:03
  • 2
    $\begingroup$ @DocBrown Correct, Generative AI existed long before LLMs. $\endgroup$
    – Dr. Snoopy
    Jan 25 at 16:03
  • 3
    $\begingroup$ @Cerbrus It will plagiarize an existing story with your characters, this has been shown over and over, the NYT lawsuit just shows that it can produce word by word content from the NYT. It applies to almost everything that is generated, another example about jokes: arstechnica.com/information-technology/2023/06/… $\endgroup$
    – Dr. Snoopy
    Jan 25 at 19:18
7
$\begingroup$

The distinction between "generative" and "non-generative" AI isn't an especially useful one. A language model (to first approximation) takes a sentence as input and tells you how probable it is that a native speaker would phrase it that way. Any remotely decent translation system is going to contain a language model, either as a separate component or integrated into the translation model proper. This helps ensure that the text it outputs is grammatically correct and scans well. Without a language model, you get a word-for-word translation that won't sound natural at all.

But if you have a probability distribution over the next word given the previous words, not only can you answer how probable an existing sentence is, but you can take a series of words and sample the next word in proportion to its probability under that model. You then put that word in the context and generate a new one, and so on. Any model that can evaluate the probability of a sentence can be trivially modified to generate a sentence. Similarly, you can take any generative model and output the distribution over the next word instead of just selecting a single one; the product of these successive distributions will be the joint distribution over sentences of that length. The models are exactly the same in either case; only the user interface differs.

(Of course, when setting policies, those user interface details might actually matter, since we might want to encourage users to fix their grammar but discourage them from posting answers when they have no idea if they're correct. But those details are entirely uninteresting mathematically.)

Historically, the use of language models in things like machine translation and voice recognition are the serious uses, and using them to generate text is a neat side task that can be used to sanity check the sort of sentences it thinks are representative of the model. (Text generators can also be fun toys, of course, which is how they're being used in things like ChatGPT.)

$\endgroup$
7
  • $\begingroup$ Maybe the distinction is not clear cut, but obviously there are AI systems which clearly do not count as GenAI, and others which clearly count as such for the majority of people. In the middle, there is a grey area. Do you think systems like Google Translate belong more clearly into the "non-GenAI" catgegory, the "GenAI" category or somewhere into that grey area? $\endgroup$
    – Doc Brown
    Jan 25 at 21:19
  • 1
    $\begingroup$ @DocBrown The problem is that the classification "Generative AI" is not based on strict mathematical or engineering properties. Likewise, banning use of "generative AI" cannot be relaibly based on a model containing some mathematical or enginering element. Translation systems are, to me, generative systems. They sample from a probability distribution of outputs, and to create a translated sequence, usually reinsert what they just sampled as "written so far", just like LLMs performing text completion. The models are very similar. What is different are the task-based constraints on ground truth. $\endgroup$ Jan 25 at 22:34
  • $\begingroup$ @NeilSlater: yes, thanks. This confirms to me that Generative AI is a vague term which is commonly used for a lot more systems than the current AI ban on Stack Overflow seems to be intended for. Hence I think it needs to be clarified whenever it is used in a certain context. I miss exactly that clarification in SO's current AI policy text. $\endgroup$
    – Doc Brown
    Jan 25 at 22:49
  • 4
    $\begingroup$ @DocBrown I think focusing on a strict definition for the ban is looking at the wrong thing. What the authors of the ban want to prevent is use of low-effort machine assisted answers that have a high chance of looking authoratitve, but that contain mistakes. The ban is a rough heuristic towards that goal. Looking for loopholes or theoretical overreach, either as a poster or as a mod, is missing the point. $\endgroup$ Jan 26 at 7:22
  • 1
    $\begingroup$ @DocBrown It's not that the meaning is broad so much as that the meaning is based on "how it's used" rather than "what it is". Given that, it would probably make sense to phrase the policies as something like "Don't use language models to create semantically significant parts of your answer (as opposed to modifying an answer you wrote, without changing the semantics (e.g., via translation or grammar improvement)." Say what you don't want the users doing instead of trying to exhaustively list all the tools they might use to do it. $\endgroup$
    – Ray
    Jan 26 at 16:12
5
$\begingroup$

Actually it depends, generative models are specific kind of machine learning models. Generative often means a model that models the probability distribution of data $p(x)$, you cannot do translation using this kind of model.

But also you can build conditional generative models, where the probability of data points $x$ conditioned to some input $y$ is modeled $p(x \, | \, y)$, and in this case it is possible to perform a translation task using a generative model, where $x$ is the translated text, and $y$ is the original text.

So it all depends on the model you use for the task, if you use a (conditional ) generative model to perform translation, assuming that the model is trained for that task (like ChatGPT), then yes, translation can be done using generative models.

The concept of translation in AI/ML is more general, to just produce an output given an input, for example, CycleGAN is a generative model that translates between two sets of unpaired images, from horse to zebra and viceversa. And GANs are generative models.

About the specific question on DeepL and Google Translate, it depends on what underlying model they use, if at some point they use generative models to perform translation, then they should also be flagged as generative AI.

In general the concept of Generative AI has... deformed over time... (due to hype). Generative AI has existed long before LLMs and Difussion models like Dall-E and Stable Difussion, just look at Generative Adversarial Networks (GANs),

$\endgroup$
4
  • 2
    $\begingroup$ We're not asking if you can. We're asking if these translation tools are generative. Your answer is basically "we don't know"... Considering the accuracy of the translations, and the tendency for generative AI to generate inconsistent output, I'm fairly sure generative AI doesn't play a role in them. $\endgroup$
    – Cerbrus
    Jan 25 at 12:11
  • $\begingroup$ @Cerbrus The OP is asking more generally, if language translation can fall into the category of generative AI, and the answer is definitely Yes. $\endgroup$
    – Dr. Snoopy
    Jan 25 at 16:05
  • 1
    $\begingroup$ @Dr.Snoopy: you understood me right. Still I agree with Cerbrus on one point: your answer did not directly answer my question about common usage of the term Generative AI. However, your former comment does, thanks for clarifying. $\endgroup$
    – Doc Brown
    Jan 25 at 16:46
  • 1
    $\begingroup$ @DocBrown Good, I integrated that comment into my answer now. $\endgroup$
    – Dr. Snoopy
    Jan 25 at 19:14
2
$\begingroup$

The original question has been updated so my answer is being updated to reflect this.

The question being asked can be boiled down to just "What is it that makes something a generative AI?" I am quite familiar in my field of computer vision with Generative Adversarial Networks, but I believe after looking at Large Language Models that they operate on a similar principle. I will go over both of these.

Generative Adversarial Networks

First I will go over the GAN which I know well, though I have not implemented one in almost 8 years. In a GAN, you have 2 machine learning models, one is generative, and one is not. There is a non-generative model called the discriminator who's job is to determine if an image belongs in the dataset or not. Then you have the generative model who is considered an attacker or adversary, who is there to try to fool the discriminator, by producing an image that the discriminator thinks belongs to the dataset, but it does not.

During training we cycle between training the discriminator on the dataset until it cannot be fooled. Then we train the adversary until it can fool the discriminator. Then we add the adversary's images to the dataset and go back to the start again. We continue this process until the discriminator can no longer be trained to tell the difference any more, and the adversary wins. Now the GAN is complete and we discard the discriminator.

The GAN can be altered to have an input parameter where you can select a class, and it will attempt to create an output of that class to fool the discriminator with that class. This is an optional feature, but something many GANs try to implement.

LLM vs GAN

LLMs on the other hand looks to only have one model. If you dig into it though, you will find out there really is 2!.

Creating LLM Vector Stores

The Vector store is a model too! It is a generative model that provides an output to the final model that interprets the output. So you have a model that is trained on a large language database, and a vector store that is trained to output segments of words similar to the input it is given. Both the input you give and the input the vector store outputs is given to the regular model, and it takes that and provides an output to you.

So mechanically, what ties these two types of generative AI together? How can we create a mechanical definition? You have a normal network, either a discriminator, or interpreter, and then you apply to it a generative network, either a adversary or tokenizer(?) respectively during training. When you are done, either you keep them together in the latter case, or you discard the discriminator and publish the adversary in the former case, and you have a generative AI.

For older translation models, they likely did not have the processing power, or technology available to perform these kinds of tasks. This is a relatively new field. They would not have been able to be generative AI. They at most would be comprised of one model during both training and publishing.

Old answer continues below:

> [arXiv:2307.15208][4] states "Generative AI refers to a set of
artificial intelligence techniques and models designed to learn the underlying
patterns and structure of a dataset and generate new data points that plausibly
could be part of the original dataset."

Considering this, what we see is that a generative AI differs from a traditional AI in that it creates new information where none was before. A traditional AI might detect which paintings belong to Raphael, while a generative AI might try to create a new painting in Raphael's style.

I think your misconception here is that all neural network AIs produce information of some kind, but they do not produce new information. The information was already there in the input in some form in a traditional AI. 

 - The location of the dog was in the picture already, the neural network just pointed to it and said, "Here it is." 
 - The tune matched a Metallica song in a database already, the neural network just correctly identified it.
 - The information was already encoded in a foreign language, the neural network just decoded the information and then reencoded it into a new language of your choice.

Now if the model had instead read the foreign language, decoded the information, and used it as a prompt to write a novel in the language of your choice, the contents of which included new information never before seen in that input, now we are talking about generative AI. This is in fact what they do.

A lot of the media talks about how chat GPT can hallucinate it's answers when you ask it straight forward questions, but the truth is, it is generative AI. It is giving you answers that could have conceivably been in the answer set. They just aren't. This is what some experts mean by it hallucinating up new answers. It's pulling them out of the void like a generative AI is supposed to do. It just isn't what the average person expects this type of program to do.

In short, what the difference between a traditional AI and a generative AI is that a traditional AI learns a dataset, and tries to tell you something about the input with respect to the dataset. The generative AI takes the input and tries to produce something similar that it thinks would be from the same grouping in what it believes the dataset should be.
$\endgroup$
10
  • $\begingroup$ Strange. It did not appear to be behind a paywall for me, but you are correct in that it is a medical paper, but it is also the correct definition as well. I don't see what the issue is with the definition. arxiv.org/pdf/2307.15208.pdf $\endgroup$ Jan 25 at 23:01
  • $\begingroup$ The definition in that paper (about medical imaging) looks like it was written by authors who picked a phrasing which suits their needs in context of that paper. This is nothing I would consider to be a valid statement about the general, common use of the term GenAI in the field of AI, more one possible use of the term in the field. In that context, that is ok and surely correct, but I would not draw any conclusions about the terminolgy for translator systems from it. $\endgroup$
    – Doc Brown
    Jan 25 at 23:07
  • $\begingroup$ ... note also I never said I believed all artificial neural network systems count as GenAI, that would have been clearly nonsense. I was talking specifically about translators, where you feed one text in as input, and get a - translated - text as output. My understanding of this process is that is is not uncommon to call this "generation" (of a text in a diffferent language). $\endgroup$
    – Doc Brown
    Jan 25 at 23:16
  • $\begingroup$ As I noted in my answer, this is a misconception on your part. Generation in this case has more than one meaning. Yes, it generates text. No it does not generate new information that did not exist in the original message, at least not intentionally. Of course mistranslations do happen, and when they do, new information can be accidentally injected, but that is considered a bug, and not a feature. $\endgroup$ Jan 25 at 23:18
  • 1
    $\begingroup$ As such, the definition says that a generative AI will produce a result out of nothing that will match what it believes should be in the dataset. A translation AI does not produce a result out of nothing. It produces it from it's input. By definition, a translation AI is not generative. $\endgroup$ Jan 25 at 23:28
-1
$\begingroup$

No, they are not "generative".

As clearly explained by Wicket on the GenAI Beta site, sourced from SirBenet's answer on the same site, Google Translate isn't "generative":

output could be considered too tightly defined by the input text

These translation tools don't generate new text. They translate.
Sure, they use AI to infer meaning to result in better translations, but you can't tell them to "Translate Macbeth to Swahili"...

You'll never get more out of the tools than you put into them.

$\endgroup$
3
  • $\begingroup$ Sirbenet's answer was refering to What is generative AI (GenAI) according to this site?, where "this site" means "Gen AI meta". I am asking about the common use of the term in the field of AI. $\endgroup$
    – Doc Brown
    Jan 25 at 11:25
  • 2
    $\begingroup$ In this question here, I am asking about the general use of the term in the field of AI, not just at SE or SO. $\endgroup$
    – Doc Brown
    Jan 25 at 12:27
  • $\begingroup$ You're free to ignore my answer. $\endgroup$
    – Cerbrus
    Jan 25 at 12:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .