2
$\begingroup$

Abstract

I wish to design a neural network that will categorize messages based on criteria I have predefined. It should feature the ability to be proactively trained as it continues its lifecycle. This means a human can intervene in its categorization attempts and determine whether or not it was correct and have it adjust its weights accordingly (without having to retrain all over again).

Inputs

It is know that ALL input will follow these rules:

  1. Always of $N$ length
  2. All messages are transformed to eliminate unnecessary complexity

Here's a brief overview of how an example message might be processed.

Starting with a message $M$:

That's an interesting perspective. I think that you should consider adding more details to your point about the cat being too silly.

The text is then transformed so that extra details are removed:

thats an interesting perspective i think that you should consider adding more details to your point about the cat being too silly

Then it's converted into a vector (appending $0$ to reach length $N$) ready to be processed by the neural network:

[116, 104, 97, 116, 115, 32, 97, . . . , 0, 0, 0, 0]

Ouputs

In my network, I wish all the outputs to be weighted on how well they fit in each category. I need multiple outputs. I'm not really focusing on one particular category per-say, but how well the message fits in all of them.

Following the input $M$ I used as an example, I'd expect the outputs to look something like this after my vector has fed-forward:

Suggestive:  0.89042
Opinionated: 0.68703

The weight values for each output indicate the strength of the category in the overall message.

From message $M$:

That's an interesting perspective.

Would weigh the opinionated category as $0.68703$.

And:

I think that you should consider adding more details to your point about the cat being too silly.

Would weigh the suggestive category as $0.89042$.

Summary and Questions

I'm interested in the architectural design choices of a network that would support my feature set. The main goal is to be able to train my network to categorize messages based on pre-trained (and live-trained) data. I'd like to know things like:

  1. What type of Neural Network I should use for this purpose? I've researched LSTM & Recurrent networks; which have been mentioned to be good at processing sequences (ie. messages).
  2. What considerations should I account for when creating this network?.
  3. How can the overall network support live-training so I can tell my network when its wrong and have it 'correct' itself without having to retrain completely?
$\endgroup$
  • $\begingroup$ How do you vectorize your inputs? $\endgroup$ – JahKnows Feb 23 at 23:55
  • $\begingroup$ @JahKnows I've considered converting each character into a float (ie. getting the code point of each) and also converting each word into a float (ie. having a word database with a float value assigned to each). Whichever method works best for my case will be how I vectorize inputs. $\endgroup$ – Anilla Feb 24 at 0:10
  • $\begingroup$ have you considered some techniques like bag-of-words, tf-idf and n-grams? $\endgroup$ – JahKnows Feb 24 at 7:41
  • $\begingroup$ (along with JK) my suggestion is try out LSA 1st to learn the concepts of converting words to vectors, and its similar to NN techniques but actually somewhat conceptually more basic using linear analysis. en.wikipedia.org/wiki/Latent_semantic_analysis also there are now several high quality open source off-the-shelf NN systems for NLP (natural language processing) that can be applied without learning all the theory. $\endgroup$ – vzn Feb 24 at 15:55
  • $\begingroup$ @vzn Thanks. The goal for me is to learn everything from scratch. Is there any places you could point me towards that really break down NLP NN concepts? $\endgroup$ – Anilla Feb 24 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.