Sentence generation for limited vocabulary

Question

I need to make a sentence generator for a limited set of vocabulary (about 600 words). The requirements are:

• It must use only the words that are on the list, and never go beyond that;
• It must produce sentences of varying syntactic structures, including complex sentences, relative clauses, tenses etc (not just the basic "SVO" sentences or whatever);
• The sentences must be at reasonably meaningful and make sense at least most of the time (meaning that the sentences like "Colourless green ideas sleep furiously" shouldn't be generated).

I'd like to ask what's the best way to go about it.

The "varying sentence structures" part is the easy one: I can do the grammar.

The issue, however, is the "meaningfullness" part. So far, I reckon that in order to implement it, I would need to generate a collocations database for each word: other words can it govern, and as what arguments specifically (e.g., the verb "give" can govern animate nouns as agents and recipients, and inanimate nouns as patients; while the verb can generally only govern some speech-related nouns like "story", "words" or "truth" as patients). I should probably be able to extract this information from a corpus: I would need to use something like deeppavlov to parse the sentence structures (in order to extract the exact relationship between words), and some tool to account for irregular verbs and inflections (I would want for all forms of the same word to be treated as the same word, obviously).

However, collocations alone aren't enough to ensure meaningful sentences: even if every clause consists of only meaningful collocations, it still doesn't prevent, say, unrelated clauses to co-occur in the same complex sentence ("I have to go to school because my dog is brown", or whatever). So in addition to a collocations database, I think I would also need a co-occurrence database, essentially telling how likely is each of the 600 words to co-occur with each of the other 599 words in the same sentence, regardless of their syntactic relationship. Basically, a total of 360000 probability values.

I'd like to ask for advice. Am I missing anything, is there perhaps an easier way to go about what I want to do?

Also, are there perhaps any existing solutions for my requirements? I briefly tried tweaking with the ChatGPT for a while, but getting it to stick to the list of words and at the same time produce diverse sentences had driven me insane pretty quickly.

P.S. So far, I'm kinda leaning towards a non-neural-network solution because of the small vocabulary list.

I'll appreciate any help or advice.

Answer 1

Building a sentence generator with a limited vocabulary looks like a challenging task, but I think it's definitely feasible. At least, while there may not be an out-of-the-box solution tailored to your exact requirements, you can leverage existing libraries and tools to simplify the development process. For instance, those like NLTK, SpaCy, or Stanford CoreNLP (which I've never used) provide various linguistic resources, including POS taggers, parsers, and syntactic information, which can help you build the necessary components. Anyway, this is what I would do:

1. Vocabulary and Sentence Structures: You already mentioned that generating varying sentence structures is relatively straightforward, so we won't focus on that.

2. Collocations Database: Creating a collocations database for each word is a good idea. This database should capture the possible arguments and dependencies for each word in your limited vocabulary. Extracting this information from a corpus, as you suggested, can be achieved using tools like Part-of-Speech (POS) taggers and dependency parsers. These tools can help identify the relationships between words in sentences and extract the required information. You can use libraries such as NLTK or SpaCy for this purpose.

3. Co-occurrence Database: To address the issue of unrelated clauses co-occurring, you can create a co-occurrence database. This database should store the likelihood of each word in your limited vocabulary co-occurring with every other word. You can calculate these probabilities from a large corpus by counting the occurrences of word pairs in the same sentence or context. The values in the database will represent the probabilities of co-occurrence.

Here's a high-level architecture diagram for your sentence generator:

   +------------------------+
   |   Collocations Database| 
   +------------------------+
                |
                v
   +------------------------+
   |  Co-occurrence Database|
   +------------------------+
                |
                v
   +------------------------+
   |   Sentence Generator   |
   +------------------------+
                |
                v
   +------------------------+
   |     Syntactic Rules    |
   +------------------------+

In my diagram above, 1. the collocations database stores information about the possible arguments and dependencies for each word in your limited vocabulary. After, 2. the co-occurrence database stores the probabilities of each word co-occurring with every other word. Then, 3. the sentence generator utilizes the collocations and co-occurrence information to generate sentences. Finally, 4. the syntactic rules component incorporates the syntactic structures and grammar rules to ensure proper sentence formation. To generate a sentence:

In details, 1) you choose a starting word from your limited vocabulary. After, 2) Use the collocations database to select the next word based on the possible arguments and dependencies. Then, 3) you utilize the co-occurrence database to determine the probability of co-occurrence between the current word and potential next words. Last, 4) you select the next word based on the highest probability from the co-occurrence database. Repeat steps 2-4 until you reach the desired sentence length or a suitable stopping condition.