Philosophically speaking,
I believe a perspective shift is necessary. When you say, 'A solution exists', the next statement should be, 'Well, what is the problem?'. While you are correct in pointing out instances where the model is unique and not so much so, their correct problem statement is not being talked about.
I am afraid, that the concept of a "Well-posed problem" is theoretical, and while a problem might look absolutely well-posed in theory, things might work out differently in the real world.
Let's talk about the theory of an LLM (will intentionally leave out the technical bits). Ideally, given an exhaustive dataset of all the world's data and a NN with practically infinite training parameters; a trained language model should be able to be attentive enough to completely understand the context before providing any answers. Such a model will work something like this-
User: "Fill in the blank, 'I would love to have some ____'"
LLM: "What do you love"
User: "Among many things, chocolate, ice-creams, dance, booze, etc."
LLM: "Okay, and how are you feeling today?"
User: "Particularly hot, and alone."
LLM: (to itself: maybe some sugar to lift up the mood, and some cold ice cream to help with the hot day)
LLM: Let's go have some Chocolate Ice-cream.
User: Wow! I would never have thought about it myself! Thanks, Overlord.
Strictly theoretically speaking, the Freudian concept of pan-determinism says, given the complete history of a human being, you can pinpoint the exact decision they will make given a situation. So, in this ideal world, where the model thought this information is enough to figure out the answer would be absolutely right. As there will exist only one fill to the blank.
Now, let's travel to the real world, shall we?
Here, everything is limited. From low memory, and limited silicon chip supply, to limited attention spans (of LLMs of course). Because here, our LLM doesn't have (yet) the capability to reason on itself, unless explicitly advised to do so, it will answer based on defined heuristics like Greedy Search (see, now that you think of it, because someone included greedy search in the architecture, suddenly it feels like it was a well-defined problem after all? Because given the limitations of the language models and the architecture set bu the creature, there is only one solution, isn't there? Unless of course, you decide you want to increase the randomness in selection). But, back to the point, transformers or any other kind of NN are essentially numerical models which theoretically speaking have a well-defined convergence to a point but practically never converge; we always judge the trained parameters to be good enough and stop the training.
Given that we can probably never really achieve the ideal doesn't mean that it doesn't exist. With infinite resources, we can build the ideal LLM that can do things that make it a Well-posed problem.
And in due time we might have LLMs good enough to predict human decisions which makes you feel like they only had one solution which they duly produced, but as is with illusions, they are never real.
UPDATE: One more thing that I just recalled is that while it is a well-posed problem, engineers and developers go to great lengths to disturb its "pose" just because again, practicality supersedes idealism. What I mean by that is illustrated best using an example, that of the Bi-directional Encoder Representations from Transformers or the BERT architecture.
The trainers went to great lengths to intentionally introduce errors in training to avoid overfitting and achieve higher efficiency, which on the downside had the side-effect of not predicting a specific token with high confidence. What I am referring to is their use of masking [Mask] only 80% of the time during training, and intentionally using [word] instead of a mask, or a [wrong word] instead of the mask equally over the remaining 20% of the times.
