I have not studied machine learning or AI really, but my job sometimes requires me to automate stuff. Right now the requirement I have, seems to be under AI domain, but I am not sure about terminologies or how to go about it. I will really appreciate if someone can guide me about the direction I need to start from.
(PS: This question might not belong on this SE, in that case please direct me to suitable SE)

What I'm required to do is find references on web about a certain situation. As an example I'll use "Music", so I have to make a system which will search around the web (Google and Twitter mainly) to see if there is any news/mention/event related to Music that occurred today, if so how many references (i.e. how big of a deal it is making).
It is not the generic term music which is expected in the output, but the names of Musicians, i.e. in Music this and this Artist appeared this many times.
I have to give the number of references, and also provide the references in output so that one can read them in detail.

The challenges are
- One event can be covered by many websites, and there can be one main website that published the original story with full details, while others just spread the word around in summarized way.
How do you filter references to pick the most suitable one, to show in results to the system's user, because I can not give user ~50 references to manually read through, I have to give like 1-2 suitable reference
- I need to give the name of the artist. One site will have many words, how do I know which word is actually the artist's name? One option can be to have a pre compiled list of specific artists and just search for them individually. But this way, I can be missing new artists.

The challenges I have, must have been addressed by some existing algorithm or mechanism, I'll appreciate if someone can let me know what kind of algo etc I need to refer to or study to get the task done.

  • $\begingroup$ Hey you,staff is a bit complex thats if you're planning to build a system perhaps a web bot[web crawler]without knowledge or background in machine learning or data mining algorithms like associative and predictive algorithms.Also Naïve Bayes Classification;this can be applied to mark a post as spam or not Spam.I conclude by saying you can't do it alone...what you want to build is not like learning a prog' lang from the tutorial.You have to have a glimpse of staff on your finger nails. $\endgroup$
    – quintumnia
    May 3 '17 at 9:32

A parallel situation might be that of spam/not spam. The detection of spam by AI has been pretty successful, so there is an existing algorithm - classification. However while you have a possible approach you are still missing the key ingredient which is sufficient data to train the model on.

AI depends on a large amount of data to train the model. Ideally you will have a team of researchers available to read a (large?) number of sources and classify by hand whether the source is the original reference or just a repeater, and whether the topic is relevant. The more labelled and balanced data you have, the better the model and the better the results. Then, like the spam/not spam situation, you just apply your model and the best references pop out. You already have the key word of music, so you just use as candidates those sources that reference "music" and any other defining keywords.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.