How do I design a neural network that breaks a 5-letter word into its corresponding syllables?

Question

I am going to design a neural network which will be able to break a 5-letter word into its corresponding syllables (hybrid syllables, I mean it will not strictly adhere to grammatical syllable rules but will be based on some training sets I provide).

Example: "train" -> "tra"-"in".

I think of implementing it in terms of some feedforward neural network with 1 input layer, 1 hidden layer and 1 output layer.

There will be 5 input nodes in the form of decimals (1/26 = 0.038 for 'A' ; 2/26 = 0.076 for 'B', etc.)

The output layer consists of 4 nodes, which corresponds to each gap between two characters in the word.

And fires as follows:

• For "train" ("tra"-"in"), the input is (0.769, 0.692, 0.038, 0.346, 0.538) and the output would be (0, 0, 1, 0).

• For "boric" ("bo"-"ri"-"c"), the input is something else, and the output is (0, 1, 0, 1).

Is it possible to implement the neural network in the way I am doing? If possible, then how will I decide the number of hidden layers and nodes in each layer?

In the book I am reading, the XOR gate problem and its implementation using hidden layer is given. In XOR, we could decide the number of nodes and hidden layers required by seeing the linear separability of XOR using two lines. But here I think such analysis can't be made..

So, how do I proceed? Or is it a trial and error process?

Answer 1

I would highly recommend modeling things differently with regard to how letters are presented to the model. While the problem is more natural, perhaps, for a Convolutional or Recurrent Neural Network, there's no problem to try and run this on a feed forward network. However, the way you give letters as input will be very confusing for a network and will make learning very hard. I'd recommend using one hot encoding or even a binary encoding for the letters. If this is for more than playing around I'd try and add some info (encode whether the letter is in "aeiou" in a separate bit).

As for the hidden layers, try playing around a bit. Two systematic approaches are to start very simple and make the model more complicated, or start complicated and make your model simpler (or just normalize a lot). Look at the performance on the training set and on a separate validation set during training. If the model keeps on improving on the training data but starts to deteriorate on the validation data, you're probably over fitting. In this case you should either make the model simpler (fewer nodes, fewer layers) or regularize (start with l2 normalization on the weights). If the data doesn't perform well on the training data, you may wish to make the model more complex.

Once you've tried the feedforward network, really do try CNN or RNNs for this task.