Assumptions
Different model structures encode different assumptions - while we often make simplifying assumptions that aren't strictly correct, some assumptions are more wrong than others.
For example, your proposed structure of "just pass the $X$ number of letters leading up to the last letter into an FFNN" makes an assumption that all the information relevant for the decision is fully obtainable from the $X$ previous letters, and $(X+1)$st and earlier input letters are not relevant - in some sense, an extension of the Markov property. Obviously, that's not true in many cases, there are all kinds of structures where long term relationships matter, and assuming that they don't lead to a model that intentionally doesn't take such relationships into account. Furthermore, it would make an independence assumption that the effect of $X$th, $(X-1)$st and $(X-2)$nd elements on the current output is entirely distinct and separate, you don't make an assumption that those features are related, while in most real problems they are.
The classic RNN structures also make some implicit assumptions, namely, that only the preceding elements are relevant for the decision (which is wrong for some problems, where information from the following items is also required), and that the transformative relationship between the input, output and the passed-on state is the same for all elements in the chain, and that it doesn't change over time; That's also not certainly true in all cases, this is quite a strong restriction, but that's generally less wrong than the assumption that the last $X$ elements are sufficient, and powerful true (or mostly true) restrictions are useful (e.g. the No Free Lunch Theorem applies) for models that generalize better; just like e.g. enforcing translational invariance for image analysis models, etc.