9
$\begingroup$

I wanted to started experimenting with neural network and as a toy problem I wished to train one to chat, i.e. implement a chatting bot like cleverbot. Not that clever anyway.

I looked around for some documentation and I found many tutorial on general tasks, but few on this specific topic. The one I found just exposed the results without giving insights on the implementation. The ones that did, did it pretty shallowy (the tensorflow documentation page on seq2seq is lacking imho).

Now, I feel I may have understood the principle more or less but I'm not sure and I am not even sure how to start. Thus I will explain how I would tackle the problem and I'd like a feedback on this solution, telling me where I'm mistaken and possibly have any link to detailed explainations and practical knowledge on the process.

  1. The dataset I will use for the task is the dump of all my facebook and whatsapp chat history. I don't know how big it will be but possibly still not large enough. The target language is not english, therefore I don't know where to quickly gather meaningful conversation samples.

  2. I am going to generate a thought vector out of each sentence. Still don't know how actually; I found a nice example for word2vec on deeplearning4j website, but none for sentences. I understood how word vectors are built and why, but I could not find an exhaustive explaination for sentence vectors.

  3. Using thought vectors as input and output I am going to train the neural network. I don't know how many layers it should have, and which ones have to be lstm layers.

  4. Then there should be another neural network that is able to transform a thought vector into a sequence of character composing a sentence. I read that I should use padding to make up for different sentence lengths, but I miss how to encode characters (are codepoints enough?).

$\endgroup$
7
$\begingroup$

I would recommend to start by reading this blogpost. You can probably cannibalise the code to create a RNN that takes in one statement of a dialogue and then proceeds to output the answer to that statement.

That would be the easy version of your project, all without word vectors and thought vectors. You are just inputting characters, so typos don't need to concern you.

The next more complex step would be to input word vectors instead of characters. That would allow you to generalise to words that aren't part of your training data. And it is probably still just a minor modification of the code.

If you insist on using thought vectors, you should start reading up on NN translation. And probably try to get a pre-trained encoder network. Or pre-train it yourself on a large translation corpus for your language.

With your small training set the best you can do is probably massively overfit until your system recreates your training data verbatim. Using word vectors will allow your system to give the same answer to "I beat the cat today." as you gave in the training data to "I kicked the dog yesterday."

I'm not sure thought vectors will make a big difference. If you get the decoder to learn at all.

$\endgroup$
  • $\begingroup$ I already read the first article some time ago and I feel characters are not the right level of abstraction. I didn't think about going with word vectors one at a time but I guess that is an interesting and less complex way too. I'll read the paper, which is probably what I was after... $\endgroup$ – Totem Dec 21 '16 at 8:49
  • $\begingroup$ I've found a bot core to use... But I am stuck on the text generation. ai.stackexchange.com/questions/5963/… What I am wondering is despite the vectors, and the learning rate, it is still not functioning as it should... I am worried that this might be a shortcoming of the Library I am using, but I do not think that could be possible for a layer size of 300... Or is the amount of training required related to the size of the layer? Any help would be appreciated. Please note the library being used. $\endgroup$ – FreezePhoenix Apr 11 '18 at 13:44
0
$\begingroup$

According to your project concept,for starters,I would request you to apply some test driven development techniques. First, try to create a smaller sized database which you can use to handle a small amount of dataset,which can give a desired improvement.

That said, use that database to create organised tree inline with your data,as nodes.So if a bot starts to generate some feedback,from the database,marked with data points from your specified dataset.And thus the feedback nor answer is the way to the next node in the tree.

note : For starters,don't use your large chat history as a whole,because this is a simple task..i.e..too much inputs == Overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.