Skip to main content
Share Your Experience: Take the 2024 Developer Survey
35 votes
Accepted

Can BERT be used for sentence generating tasks?

For newbies, NO. Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its ...
soloice's user avatar
  • 511
15 votes

Why are LLMs able to reproduce bodies of known text exactly?

Google "call me ishmael. some years ago—never mind how long precisely—having" and you'll see a fair number of results. LLM training sets are likely to have several copies of it as well, ...
Franck Dernoncourt's user avatar
13 votes

Why are LLMs able to reproduce bodies of known text exactly?

LLMs are information-theoretically just very lossy compression of their entire corpora, and are large enough for the "decompression" of parts to be recognizable and reasonably faithful. I ...
R.. GitHub STOP HELPING ICE's user avatar
9 votes

Can BERT be used for sentence generating tasks?

this experiment by Stephen Mayhew suggests that BERT is lousy at sequential text generation: http://mayhewsw.github.io/2019/01/16/can-bert-generate-text/ ...
stuart's user avatar
  • 191
6 votes

How much of the ChatGPT output is copied from its training set (vs. being abstractively generated)?

From the paper Language Models are Changing AI: The Need for Holistic Evaluation (Authors: Rishi Bommasani and Percy Liang and Tony Lee; Website): Memorization of copyrighted/licensed material. We ...
Franck Dernoncourt's user avatar
3 votes
Accepted

Clarification on GANs for text generation

When you sample a sentence using an RNN, at each step $t$, if $N$ is the size of your vocabulary, $W$ the array with the $N$ words in your vocabulary, and $f$ your RNN without its final linear layer, ...
Raphael Lopez Kaufman's user avatar
3 votes
Accepted

Are there any good alternatives to an LSTM language model for text generation?

The current state of the art in natural language generation are all auto-regressive transformer models. Transformers no longer use recurrent neural networks such as LSTM, because the recurrences makes ...
user3667125's user avatar
  • 1,570
3 votes
Accepted

How to use LSTM to generate a paragraph

Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That'...
jgleoj23's user avatar
  • 182
3 votes

How do I use GPT-2 to summarise text?

In short: It depends. Where will you run it? On Premises: You may want want to run in your own environment. IaaS: GPT models are often too big, so people might prefer to setup a different server for ...
Andre Goulart's user avatar
2 votes
Accepted

Discrepancies in the Exclusion of Elements in Image vs. Text Generation

The difference is due to qualities of text training data used in training the two types of model. The LLM models are trained on large amounts of text, and learn grammar rules very well. There are lots ...
Neil Slater's user avatar
  • 32.5k
2 votes

Can BERT be used for sentence generating tasks?

No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). Because of bi-directionality of BERT, BERT cannot be used as a ...
Astariul's user avatar
  • 371
2 votes

How can GPT-3 be used for designing electronic circuits from text descriptions?

As of today (9 Feb 2023) it is possible to create a HSPICE output from verbal description of an electric circuit using chatGPT. Simply describe a circuit and its element to chatGPT, and ask it to ...
user45498's user avatar
2 votes

How to use LSTM to generate a paragraph

As you know, an LSTM language model takes in the past word and tries to predict the new one and continue over a loop. A sentence is divided into tokens and depending on different method, the tokens ...
Clement's user avatar
  • 1,745
1 vote

Dynamic text completion

That is achieved usually via web-socket, however it is not available yet I see some workaround like https://github.com/cookeem/chatgpt-service, which just get's the full answer and then streams word ...
Alberto's user avatar
  • 2,198
1 vote

Can you automatically write a text summarizing a finance document using AI models?

I think it is a quite reasonable task. To extract text from a PDF there are many libraries. If you also have images or other plots could be transformed in an embedding using a detector followed by ...
Ciodar's user avatar
  • 400
1 vote
Accepted

How can GPT-3 be used for designing electronic circuits from text descriptions?

This is rather something for V&L Models, training on associated texts with circuit images. Data which should be hard to come by. I doubt these models are yet capable of catching enough detail in ...
Langueneers's user avatar
1 vote
Accepted

Making generated texts from "data-to-text" more variable

You could handwrite different templates and choose probabilistically, according to writing style or pragmatic effects like irony and so on, but that very much depends on the domain. If you have ...
Langueneers's user avatar
1 vote

Making generated texts from "data-to-text" more variable

The state of the art in text generation is the GPT model. GPT-3, which was just released in summer of 2020, has been used to generate many very impressive articles, and is widely considered the best ...
chessprogrammer's user avatar
1 vote

Transformer Language Model generating meaningless text

This is probably an issue of complete underfitting. How many training data do you use? What is your vocab size? What is your batch size and how many epochs did you train? Transformers always need more ...
Langueneers's user avatar
1 vote

Is there a complement to GPT/2/3 that can be trained using supervised learning methods?

This is the task of so-called V&L (vision and language models) which effectively encode information from both worlds. There are also many training corpora covering this field already. Here is a ...
Langueneers's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible