Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
Many machine translation metrics such as BLEU or ROUGE are used to evaluate sequence to sequence models where, usually, the sequences are pieces of natural language.
Is it possible to use these metrics when the dataset is not constituted of natural language sequences? For instance, if the sequences are source code (in some programming language), does it still make sense to use BLEU or ROUGE? How "good" are these metrics in general?
Yes - and no. The important distinction is whether your data contains proper word boundaries and rigorous translation references.
BLEU and ROGUE both work by comparing a candidate (ie, model output) to reference text (ie, training data). In a translation task (what these metrics are typically used for) this works quite well, as you can normally assume the translation will use a small set of words common among all references, so naturally the candidate should be using these words in some order. The more references you have the more allowance the model has to interpret the translation it's own way and still achieve a relatively high score.
However, this does not always work. Both algorithms require word boundaries for them to work properly. You can modify your dataset so that words are properly separated for the algorithms to use, but using your example of source code, this becomes very tricky with indentation and more. Additionally, it's very hard to capture the possible outputs. Not only can a variable be reasonably named hundreds of different ways (making it unlikely an exhaustive reference list exists that would capture your models output), the amount of potentially correct code grows extremely quickly the larger the task becomes, meaning the references become increasingly less useful. If you keep the tasks very short, you could potentially find some use from these algorithms, but better metrics exist.
It's probably best to use these algorithms for what they're meant for - translation. However, there is absolutely no harm in using them in a task anyway, they're very simple algorithms and could still potentially give you some insight.
(For source code evaluation, you would probably be better actually just running the code and seeing if it completes the task, maybe making your own scoring system. Or you could draw inspiration from something such as OpenAI's Codex HumanEval)
