1
$\begingroup$

I'm wondering if these 2 specific programs already exist and if not how hard would it be to write them:

  1. A program that would figure out (by only "reading" large amounts of texts in human language 1 and 2) which words in second language have the same meaning as a word in first language. You would give for input texts in both languages and for output you would get for every word in first language a list of words in second language that are most similar to it with a probability that they mean the same thing.

  2. A program that would figure out which words have the most similar meaning by analyzing large amounts of texts in one human language.

I'm planning on writing these two programs and it would be nice if I could get existing programs that do this so that I could compare results of my program to those of existing programs.

|improve this question
$\endgroup$
  • 1
    $\begingroup$ A point of consideration: words are only one of many forms of structure in a language. That is to say, different languages will construct the context around synonymous words in slightly different ways. Any good model outputting a translation of sorts will almost certainly have to account for this heterogenous word context to some degree. For a great book that covers some interesting ideas around differences in mechanics between language, check out Through The Language Glass by Guy Deutscher. $\endgroup$ – Greenstick Feb 10 '18 at 16:11
  • $\begingroup$ Can you give an example of what you mean? Yes for example order of words can be different in different languages. But the way one word is related in meaning to other words is same in all languages, because languages live in the same worlds where logic is the same. $\endgroup$ – Tone Škoda Feb 10 '18 at 17:09
  • $\begingroup$ Good question different languages adopt there own words to similar situations and equally have different meanings dependent upon the context of there position within the communication and a response would have to have knowledge determined by learning the use of each within the boundaries of that particular context and is not simply a determination absolute and singularly to the meaning of a word $\endgroup$ – Bobs Feb 10 '18 at 19:17
  • $\begingroup$ That’s a peerless sentence right there! @Bobs Yes, the short of it is that the variation in grammatical constructs and the structure of relationships between words (among other things) need to be understood by your program (this is not covered in the book I mentioned). Given the complexity of what your asking, this program almost certainly has at least one statistical learning model at its core. Also, thinking of synonyms we can see how context is critical: a given synonym in language A undoubtedly does not share the same set of synonyms for its synonym in language B. $\endgroup$ – Greenstick Feb 11 '18 at 2:36
  • $\begingroup$ Don't know if statistical learning would help though but I guess there is an element of chance in a decision to accept the meaning of a word other than the one conceived inferred from the context of it's new surrounding. But recursively you would be selecting randomly from a knowledge all ready potentially to contain the correct inference so where had that come from. I think therefore there's a question of inspiration in humans which maybe not entirely chance based but in intuition not entirely down to knowledge gained by learning. $\endgroup$ – Bobs Feb 13 '18 at 21:03
1
$\begingroup$

For the first answer, the general way is using the seq2seq model:

https://www.tensorflow.org/tutorials/seq2seq

(a more specific example of the above category is this paper:

http://emnlp2014.org/papers/pdf/EMNLP2014179.pdf)

For the 2nd answer, one way is to use word2vec and apply supervised learning on the groups of similar words:

https://towardsdatascience.com/word-to-vectors-natural-language-processing-b253dd0b0817

https://spacy.io/usage/vectors-similarity

|improve this answer
$\endgroup$
1
$\begingroup$

Let us simplify case 1 of the question: assume two files, first one with numbers written in indo-arabic numeral system (i.e. 123, 9, 186754, ...), second one in roman numeral system (XX, LCVI, IX, ...). How do you match pairs of symbols with same meaning ?

Without external information or assumptions, you can not. You could made the hypothesis that probability of one specific number is the same in both examples, and base your pairing on it. But then you need to find two input files that fulfills this condition.

|improve this answer
$\endgroup$
  • $\begingroup$ It is possible to get this information. Because same words in both languages are used in sentences that have same structure. For example lion and Africa are often used in same sentence, or stars and universe and telescope. But on the other hand stars and walking are less related, you fly to stars not walk, and aeroplane also flies not walks, and people fly on aeroplanes. So there you get these specific relationship between these words that are true in both languages and specific structure of relationships is true only for these specific words. Thats how you can know they must mean the same. $\endgroup$ – Tone Škoda Feb 10 '18 at 11:03
  • $\begingroup$ How to compare these structures is not yet entirely clear to me. It is complicated. But you have to use analogies, statistics and logic. Neural networks only use analogies or simple similarity but ignore logic, that is their problem I think. $\endgroup$ – Tone Škoda Feb 10 '18 at 11:05
  • $\begingroup$ About the examples you provide: in the pure case, you can know from text1 that there are a statistic relation between "lion" and "Africa", from text2 a relation between "estrellas" and "universo". Is "estrellas" synonym of "lyon" ? You can improve the algorithm starting by some well known fixed matches (the external information I refer in m answer), i.e 5=V, and, from this one and using statistics, infers new matches like 6=VI. However, in natural language, all these approach will have a lot of troubles. Nice as theoretical exercise, but easier to use an existing dictionary. $\endgroup$ – pasaba por aqui Feb 10 '18 at 11:24
  • 1
    $\begingroup$ Remember the case of Egyptian scripts: no translation until rosetta stone (external information) was found. $\endgroup$ – pasaba por aqui Feb 10 '18 at 11:37
  • 1
    $\begingroup$ You have my best wishes in your target. Yes, when you say "I'm planning on writing these two programs", be prepared to say "I'm planning to research on this issue and implement some test programs". You will be famous if you decrypt Voynich manuscript. $\endgroup$ – pasaba por aqui Feb 10 '18 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.