Suppose I have images of hand-written Japanese text. If I want to translate those images, would my ML algorithm be a 2-step model (for example, a CNN to convert the image into Japanese characters/tokens and then feed those tokens in an RNN)? Is this normally how it would be done, or is there an end-to-end solution?
$\begingroup$
$\endgroup$
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
As far as I know, this is always done in separate steps. The reason is the availability of training data. For the recognition task (scene text recognition or handwriting recognition), the available data are monolingual.
A solution would be generating synthetic visual data from parallel corpora. Very recently, there was a paper at EMNLP 2021 that did exactly this and show high robustness towards OCR errors. However, they do not evaluate the model on real-world images and still assume two steps: recognition and translation.
There is also a domain mismatch between what is typically in handwriting and how the usual data machine translation data look like. From the machine translation perspective, this could be best approached as a domain adaptation problem (e.g, using domain-specific back-translated data). This would quite unhandy with an end-to-end recognition and translation system.
