I am preparing to perform research comparing the performance of two different systems that probabilistically generate the next word of an input sentence.
For example, given the word 'the', a system might output 'car', or any other word. Given the input 'the round yellow', a system might output 'sun', or it might output something that doesn't make sense.
My question is, how can I quantitatively evaluate the performance of the two different systems performing this task? Of course if I tested each system manually I could qualitatively determine how often each system responded in a way that makes sense, and compare how often each system responds correctly, but I'd really like a meaningful quantitative method of evaluation that I could preferably automate.
Precision and recall don't seem like they would work here, seeing as for each given input there are many potentially acceptable outputs. Any suggestions?