1
$\begingroup$

Suppose I have images of hand-written Japanese text. If I want to translate those images, would my ML algorithm be a 2-step model (for example, a CNN to convert the image into Japanese characters/tokens and then feed those tokens in an RNN)? Is this normally how it would be done, or is there an end-to-end solution?

$\endgroup$
1
  • $\begingroup$ Here is a related question. $\endgroup$
    – nbro
    Dec 12, 2021 at 11:23

1 Answer 1

0
$\begingroup$

As far as I know, this is always done in separate steps. The reason is the availability of training data. For the recognition task (scene text recognition or handwriting recognition), the available data are monolingual.

A solution would be generating synthetic visual data from parallel corpora. Very recently, there was a paper at EMNLP 2021 that did exactly this and show high robustness towards OCR errors. However, they do not evaluate the model on real-world images and still assume two steps: recognition and translation.

There is also a domain mismatch between what is typically in handwriting and how the usual data machine translation data look like. From the machine translation perspective, this could be best approached as a domain adaptation problem (e.g, using domain-specific back-translated data). This would quite unhandy with an end-to-end recognition and translation system.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .