Which approaches could I use to create a simple chatbot using a neural network?

Question

I wanted to start experimenting with neural networks, so I decided to make a chatbot (like Cleverbot, which is not that clever anyway) using them.

I looked around for some documentation and I found many tutorials on general tasks, but few on this specific topic. The one I found just exposed the results without giving insights on the implementation. The ones that did, did it pretty shallowly (the TensorFlow documentation page on seq2seq is lacking, IMHO).

Now, I feel I may have understood the principle more or less, but I'm not sure and I am not even sure how to start. Thus I will explain how I would tackle the problem and I'd like feedback on this solution, telling me where I'm mistaken, and possibly have any link to detailed explanations and practical knowledge on the process.

1. The dataset I will use for the task is the dump of all my Facebook and WhatsApp chat history. I don't know how big it will be but possibly still not large enough. The target language is not English, therefore I don't know where to quickly gather meaningful conversation samples.

2. I am going to generate a thought vector out of each sentence. Still don't know how, actually; I found a nice example for word2vec on the deeplearning4j website, but none for sentences. I understood how word vectors are built and why, but I could not find an exhaustive explanation for sentence vectors.

3. Using thought vectors as input and output I am going to train the neural network. I don't know how many layers it should have, and which ones have to be LSTM layers.

4. Then there should be another neural network that is able to transform a thought vector into a sequence of characters composing a sentence. I read that I should use padding to make up for different sentence lengths, but I miss how to encode characters (are codepoints enough?).

Answer 1

I would recommend to start by reading this blogpost. You can probably cannibalise the code to create a RNN that takes in one statement of a dialogue and then proceeds to output the answer to that statement.

That would be the easy version of your project, all without word vectors and thought vectors. You are just inputting characters, so typos don't need to concern you.

The next more complex step would be to input word vectors instead of characters. That would allow you to generalise to words that aren't part of your training data. And it is probably still just a minor modification of the code.

If you insist on using thought vectors, you should start reading up on NN translation. And probably try to get a pre-trained encoder network. Or pre-train it yourself on a large translation corpus for your language.

With your small training set the best you can do is probably massively overfit until your system recreates your training data verbatim. Using word vectors will allow your system to give the same answer to "I beat the cat today." as you gave in the training data to "I kicked the dog yesterday."

I'm not sure thought vectors will make a big difference. If you get the decoder to learn at all.