# Is my approach to building an RNN to predict the probability that the word is in English appropriate?

## Goal

To build an RNN which would receive a word as an input, and output the probability that the word is in English (or at least would be English sounding).

Example

input:  hello
output: 100%

input:  nmnmn
output: 0%


## Approach

Here is my approach.

### RNN

I have built an RNN with the following specifications: (the subscript $$i$$ means a specific time step)

The vectors (neurons):

$$x_i \in \mathbb{R}^n \\ s_i \in \mathbb{R}^m \\ h_i \in \mathbb{R}^m \\ b_i \in \mathbb{R}^n \\ y_i \in \mathbb{R}^n \\$$

The matrices (weights): $$U \in \mathbb{R}^{m \times n} \\ W \in \mathbb{R}^{m \times m} \\ V \in \mathbb{R}^{n \times m} \\$$

This is how each time step is being fed forward:

$$y_i = softmax(b_i) \\ b_i = V h_i \\ h_i = f(s_i) \\ s_i = U x_i + W h_{i-1} \\$$ Note that the $$+ W h_{i-1}$$ will not be used on the first layer.

### Losses

Then, for the loss of each layer, I used cross entropy ($$t_i$$ is the target, or expected output at time $$i$$): $$L_i = -\sum_{j=1}^{n} t_{i,j} \ln(y_{i,j})$$

Then, the total loss of the network: $$L = \sum L_i$$

### RNN diagram

Here is a picture of the network that I drew: ### Data pre-processing

Here is how data is fed into the network:

Each word is split into characters, and every character is split into a one-hot vector. Two special tokens START and END are being appended to the word from the beginning and the end. Then the input at each time step will be every sequential character without END, and the output at each time step will be the following character to the input.

Example

Here is an example:

2. Split it into characters and append the special tags: START c a t END
3. Transform into one-hot vectors: $$v_1, v_2, v_3, v_4, v_5$$
4. Then the input is $$v_1, v_2, v_3, v_4$$ and the output $$v_2, v_3, v_4, v_5$$

### Dataset

For the dataset, I used a list of English words.

Since I am working with English characters, the size of the input and output is $$n=26+2=28$$ (the $$+2$$ is for the extra START and END tags).

### Hyper-parameters

Here are some more specifications:

• Hidden size: $$m=100$$
• Learning rate: $$0.001$$
• Number of training cycles: $$15000$$ (each cycle is a loss calculation and backpropagation of a random word)
• Activation function: $$f(x) = \tanh(x)$$

## Problem/question

However, when I run my model, I get that the probability of some word being valid is about 0.9 regardless of the input.

For the probability of a word begin valid, I used the value at the last layer of the RNN at the position of END tag after feeding forward the word.

I wrote a gradient checking algorithm and the gradients seem to check up.

Is there conceptually something wrong with my neural network?

I played a bit with $$m$$, the learning rate, and the number of cycles, but nothing really improved the performance.

• Does the training data include non-words too, or only words? – Tanner Swett Mar 22 at 18:48
• Only words. I saw a similar RNN online that used a database of sentences to generate new ones, and they didn't use "fake sentences", so I thought using only real words in mine would be enough. – Roy Varon Mar 22 at 19:55

while using a neural network for this type of problem is not the ideal use-case, it is a good exercise.

In terms of conceptual issues, the most concerning that I see is the loss: $$\sum_{i=1}^N L_i$$.

First issue, is that it validates loss at each time step equivalently. This is probably not ideal because in the example (cat), we dont expect it to know its English from just c or ca, but at cat. The quickest fix and probably the best, is to just use $$L = L_N$$. Though an argument would be that the model should become more and more aware as it gets more letters, and this loss function doesnt achieve that, so another solution would be to add another fixed parameter that you can play with: $$L = \sum_i r^{-i}L_i$$ where $$0 \lt r \lt 1$$. Note that $$r^{-i}$$ can be replaced with any function that increases in size.

Another issue with the loss is it doesnt normalize, meaning on average, larger words will hold more weight to the model than smaller ones, while this may be intended it should be noted and considered (also note if you do end up going with $$L=L_N$$ this will no longer be a concern.

Hope this helps

• Thank you for your answer. I just tried implementing your modification and the performance of the network seemed to improve a bit. Still not perfect, but modifying the loss function was definitely a step in the right direction. Btw, splitting the word into characters, and feeding them one by one to the network was not meant for testing if each substring is a valid word, but if a substring is a likely sequence of characters, for example: is it probable for the letter "t" to come after the sequence "ca". – Roy Varon Mar 21 at 18:35
• are labels done that way? ex: flys -> 0011 – mshlis Mar 21 at 19:37
• Almost, they are encoded using one-hot vectors. ex: flys -> [onehot(START), onehot('f'), onehot('l'), onehot('y'), onehot('s'), onehot(END)] – Roy Varon Mar 21 at 22:30
• i was asking about the labels – mshlis Mar 21 at 22:31
• I don't think I understand. There are no labels. There is just a database of English words. – Roy Varon Mar 21 at 22:32

I think the problem is that you're only training the network on words. Every example in your training data has a desired label of "is a word," and so your network could achieve the lowest possible loss by simply giving a probability of 100% to "is a word" all of the time.

The most straightforward way to fix this would be to also include non-words in your training data. Of course, the words should have a target label of "is a word" and the non-words should have a target label of "is not a word."