I'm trying to train an AI using reinforcement learning to play the game Wordle.

The way the game works is that there is a secret five letter code word that you need to guess, and every time you guess a word (it has to be an actual word it can't just be any five letters) the game tells you what letters of your guess are correct and what letters of your guess are in the code word but not in the correct position. If you guess the word in six guesses or less, you win.

I want every step the AI to choose a string from a list of valid guesses. Afterwards I will reward it for discovering letters for the first time and discovering their position for the first time. It will be penalized for losing and rewarded for winning based on how early it won.

The problem is that I can't just number each word and have it choose a number (I think?) because the actual content of the string it chooses matter.

How should I tackle this issue?

One thing that might be worth mentioning is that while I'm aware this could be solved algorithmically without the use of reinforcement learning, I want to try this approach anyway. This is less about the finale product and more of an excuse to get familiar with machine learning.

  • $\begingroup$ re: "This is my first machine learning project", you can most likely write a strong Wordle bot without using ML. The ML component is likely to be complex due to the action space, as you've noticed. If learning ML/RL is the goal, you will be a lot better off picking a smaller challenge. $\endgroup$ Feb 11, 2022 at 21:56
  • $\begingroup$ For people not familiar with this game, you need to explain how the "content of the string matters". Moreover, there's not description of how you're using RL here. $\endgroup$
    – nbro
    Feb 12, 2022 at 8:44
  • $\begingroup$ @nbro thanks for the feedback. I added those to the post. $\endgroup$
    – Yuval Amir
    Feb 12, 2022 at 10:31
  • $\begingroup$ I have heard of, but not played Wordle. This fact is important to possible strategies when designing an ML component: Could you explain whether Wordle only accepts valid 5-letter words as input, or if it will accept and score non-words like "ABCDE"? $\endgroup$ Feb 12, 2022 at 12:26
  • $\begingroup$ @NeilSlater It only accepts five letter words and you can't input gibberish it checks if the word is in a list of accepted answers. $\endgroup$
    – Yuval Amir
    Feb 12, 2022 at 12:28

1 Answer 1


A quite effective way to solve Wordle is to maintain an active list of all five-letter words that meet criteria so far, choose one of them completely at random, then update the list of possible words based on new knowledge from the response.

A slightly smarter way would be to pick words from the subset of remaining ones that should maximise information gathered from the response on average. I believe I read a headline recently that "crane" is a good starting English word for that reason.

The trouble for you here is that those approaches do not involve ML in any form. Relatively simple logic and analysis are already pretty good at playing Wordle automatically.

However, to stick with the question as asked, your problem is how to get a machine learning algorithm to choose between maybe 10,000 options.

You have a couple of choices:

  • Brute force. Number the options, have the algorithm choose directly. If you do this, I recommend that you also filter out logically impossible words, as that makes the problem far easier. The 10,000 options may seem large at first, but actually this should be within reach of a personal/hobby setup. It will work mainly because it is possible to simulate the results of games very fast. So generating the training data within RL will be fast, the agent can get to see millions of games in a reasonable time frame, training with each target word thousands of times on average.

  • Actions represent additional constraints on word choice. For a simple example, you could have the agent decide whether the first unknown letter it wants to try is a vowel or consonant = just two actions. You can use that to further restrict word choice from the remaining words. This influences an otherwise random word selector, and the agent should learn situations in which to pick words with certain traits.

  • Actions describe a "word vector", and the selector picks the closest valid word to that vector, breaking ties randomly. In order to do this, you need to generate some short vector of each word in the dictionary. For a simple example* you might choose [part_of_speech, popularity] to qualify each word, scoring part_of_speech as 1 for noun, -1 for verb and 0 for everything else and popularity as 1 for common word, -1 for rare word, and 0 for moderately frequent. Then your action output would be a choice from the 9 combinations possible in this vector.

None of these approaches are simple, so not great if you are just starting in AI/ML, and I would recommend you tackle a simpler toy problem or two using RL before trying to write your Wordle training routine. Going straight to Wordle without solving smaller problems first is likely to cause a lot of incorrect decisions on your part that you won't spot and will frustrate you.

* I have given a very simple example, but this could logically lead to learned word embeddings, which are an important topic in natural language processing. These are typically quite large real-valued vectors though (maybe 64 or 128 elements) which would be an even larger space to learn about than the fixed list of words. So full word embeddings like word2vec and Glove are not really suitable for the Wordle puzzle.

  • $\begingroup$ Thanks for the suggestions! I'm not sure what you mean by "logically impossible words" though. $\endgroup$
    – Yuval Amir
    Feb 12, 2022 at 13:22
  • $\begingroup$ @YuvalAmir: After you choose your first word "hello" you may know that the word contains an "e" (but not second position) and the "l" in fourth position is correct. That also implies "h" and "o" are not in the target word. So you can immediately remove all words from consideration that contain "h" or "o", that have "e" in second position, that have no "e" at all (because there is at least one), and that don't have "l" in fourth position. So you can keep "cable" and "eagle" in the list, but not "helps" because the latter is inconsistent with the new feedback. $\endgroup$ Feb 12, 2022 at 13:32
  • $\begingroup$ oh I get it. Now that I think about it, could the idea of the "word vector" be used by having the ai describe what letters it wants and you find the closest fitting word? $\endgroup$
    – Yuval Amir
    Feb 12, 2022 at 13:35
  • $\begingroup$ @YuvalAmir Yes it could, although overall that is an even larger space than 10,000 words, it may be simpler to code and be reasonably effective. $\endgroup$ Feb 12, 2022 at 13:36
  • $\begingroup$ "pick words from the subset of remaining ones": the optimal strategy has not this constraint, just to recover as much information as possible (that is, test a word that splits the set of valid words in equal sized sets). $\endgroup$ Feb 13, 2022 at 16:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .