Is my approach to building an RNN to predict the probability that the word is in English appropriate?

Question

Goal

To build an RNN which would receive a word as an input, and output the probability that the word is in English (or at least would be English sounding).

Example

input:  hello 
output: 100%

input:  nmnmn 
output: 0%

Approach

Here is my approach.

RNN

I have built an RNN with the following specifications: (the subscript $i$ means a specific time step)

The vectors (neurons):

$$ x_i \in \mathbb{R}^n \\ s_i \in \mathbb{R}^m \\ h_i \in \mathbb{R}^m \\ b_i \in \mathbb{R}^n \\ y_i \in \mathbb{R}^n \\ $$

The matrices (weights): $$ U \in \mathbb{R}^{m \times n} \\ W \in \mathbb{R}^{m \times m} \\ V \in \mathbb{R}^{n \times m} \\ $$

This is how each time step is being fed forward:

$$ y_i = softmax(b_i) \\ b_i = V h_i \\ h_i = f(s_i) \\ s_i = U x_i + W h_{i-1} \\ $$ Note that the $ + W h_{i-1}$ will not be used on the first layer.

Losses

Then, for the loss of each layer, I used cross entropy ($t_i$ is the target, or expected output at time $i$): $$ L_i = -\sum_{j=1}^{n} t_{i,j} \ln(y_{i,j}) $$

Then, the total loss of the network: $$ L = \sum L_i $$

RNN diagram

Here is a picture of the network that I drew:

Data pre-processing

Here is how data is fed into the network:

Each word is split into characters, and every character is split into a one-hot vector. Two special tokens START and END are being appended to the word from the beginning and the end. Then the input at each time step will be every sequential character without END, and the output at each time step will be the following character to the input.

Example

Here is an example:

1. Start with a word: "cat"
2. Split it into characters and append the special tags: START c a t END
3. Transform into one-hot vectors: $v_1, v_2, v_3, v_4, v_5$
4. Then the input is $v_1, v_2, v_3, v_4$ and the output $v_2, v_3, v_4, v_5$

Dataset

For the dataset, I used a list of English words.

Since I am working with English characters, the size of the input and output is $n=26+2=28$ (the $+2$ is for the extra START and END tags).

Hyper-parameters

Here are some more specifications:

• Hidden size: $m=100$
• Learning rate: $0.001$
• Number of training cycles: $15000$ (each cycle is a loss calculation and backpropagation of a random word)
• Activation function: $f(x) = \tanh(x)$

Problem/question

However, when I run my model, I get that the probability of some word being valid is about 0.9 regardless of the input.

For the probability of a word begin valid, I used the value at the last layer of the RNN at the position of END tag after feeding forward the word.

I wrote a gradient checking algorithm and the gradients seem to check up.

Is there conceptually something wrong with my neural network?

I played a bit with $m$, the learning rate, and the number of cycles, but nothing really improved the performance.

Answer 1

while using a neural network for this type of problem is not the ideal use-case, it is a good exercise.

In terms of conceptual issues, the most concerning that I see is the loss: $\sum_{i=1}^N L_i$.

First issue, is that it validates loss at each time step equivalently. This is probably not ideal because in the example (cat), we dont expect it to know its English from just c or ca, but at cat. The quickest fix and probably the best, is to just use $L = L_N$. Though an argument would be that the model should become more and more aware as it gets more letters, and this loss function doesnt achieve that, so another solution would be to add another fixed parameter that you can play with: $L = \sum_i r^{-i}L_i$ where $0 \lt r \lt 1$. Note that $r^{-i}$ can be replaced with any function that increases in size.

Another issue with the loss is it doesnt normalize, meaning on average, larger words will hold more weight to the model than smaller ones, while this may be intended it should be noted and considered (also note if you do end up going with $L=L_N$ this will no longer be a concern.

Hope this helps

Answer 2

I think the problem is that you're only training the network on words. Every example in your training data has a desired label of "is a word," and so your network could achieve the lowest possible loss by simply giving a probability of 100% to "is a word" all of the time.

The most straightforward way to fix this would be to also include non-words in your training data. Of course, the words should have a target label of "is a word" and the non-words should have a target label of "is not a word."