I have a text generation model and I want to evaluate its output by comparing it to a set of gold human-annotated references.
I went through machine-translation metrics and I found that BLEU is used as the main metric usually. I didn't like using it because it's shallow as it uses ngrams comparison; the semantics of the translation is missed.
Is there any other metric to do a semantic-based evaluation?
I've thought of using a text similarity model to evaluate the output or even an NLI (Natural language inference) system. I am not sure how precise the evaluation will be because SOTA systems are not really accurate.