1
$\begingroup$

Assume somebody knows only to write in Latin characters. If they write words of any other language (example: Hindi, French, Latin) using the Latin alphabet, how can I detect that language?

Example: If they write Hindi language word using the Latin alphabet)

kya kar raha hai

     >> the output is Hindi language
$\endgroup$
1
  • 1
    $\begingroup$ You need to distinguish between alphabet (Latin, Cyrillic, Greek, Devangari, ...) and language (English, French, Hindi). $\endgroup$ Dec 6, 2019 at 11:48

1 Answer 1

2
$\begingroup$

The most distinct words of a language are usually the function words (the, and, of, with,...); other lexical items are often (at least partly) shared between languages that had come in contact with each other. So looking for function words is usually the best way to identify the language in a given text.

This can be done by having a list of function words for each language you recognise, look those words up in your text, and then calculate the probability that the text belongs to each language given the frequencies of the words in your list. There will be overlaps, for example Dutch of is equivalent to the conjunction or in English, but also looks like the English of.

Another approach would be to take texts in your languages, and split them into n-grams, typically trigrams. Then you use a list of trigrams and compare the frequency distributions in your known and unknown texts to find which is most likely to match. This has the advantage that you don't need to know anything about the language structure (because you don't need to identify function words), and it also captures the morphology (eg the common English suffix -ing).

You can simply implement this in a fairly basic program, or you could train a machine learning classifier with it. The former seems easier to me, but is not as exciting as the latter.

However, for single words (as per your example) this might not work properly. Generally, the longer the text, the more precise the recognition.

UPDATE: On re-reading your question I think I have misunderstood it, and that you're asking about transliteration into Latin characters. In a way the same applies, that you need to first identify which language it is, and then choose the correct mapping from the Latin transliteration to the writing system that language natively uses.

Since the transliteration is purely symbolic, I don't think that an ML approach would be better than a simple lookup-table of equivalences. And of course there might be difficulties if the writing system has more distinct characters than the Latin alphabet provides.

UPDATE 2: If the input is transliterated text, then your reference texts (where you know the language) also have to be transliterated in the same character set (eg Latin in your example).

$\endgroup$
2
  • $\begingroup$ @Oliver Mason.: thanks so much for the input. I edited my question. kindly go through it. go here : translate.google.co.in/?hl=en&tab=TT&authuser=0 and type "kya", google will say its "Hindi detected". How does that happen? $\endgroup$
    – Hanu
    Dec 6, 2019 at 11:19
  • $\begingroup$ @ DuttaA : No characters. just the language is enough. we can easily convert it to hindi characters. $\endgroup$
    – Hanu
    Dec 6, 2019 at 11:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .