Understanding LSTM through example

Question

I want to code up one time step in a LSTM. My focus is on understanding the functioning of the forget gate layer, input gate layer, candidate values, present and future cell states.

Lets assume that my hidden state at t-1 and xt are the following. For simplicity, lets assume that the weight matrices are identity matrices, and all biases are zero.

htminus1 = np.array( [0, 0.5, 0.1, 0.2, 0.6] )
xt = np.array( [-0.1, 0.3, 0.1, -0.25, 0.1] )

I understand that forget state is sigmoid of htminus1 and xt

So, is it?

ft = 1 / ( 1 + np.exp( -( htminus1 + xt ) ) )

>> ft = array([0.47502081, 0.68997448, 0.549834  , 0.4875026 , 0.66818777])

I am referring to this link to implement of one iteration of one block LSTM. The link says that ft should be 0 or 1. Am I missing something here?

How do I get the forget gate layer as per schema given in the below mentioned picture? An example will be illustrative for me.

Along the same lines, how do I get the input gate layer, it and vector of new candidate values, \tilde{C}_t as per the following picture?

Finally, how do I get the new hidden state ht as per the scheme given in the following picture?

A simple, example will be helpful for me in understanding. Thanks in advance.

Answer 1

This is an image to better understand lstm... At $f_t$, we are taking the sigmoid of a weight matrix * the input at the current timestep + another weight matrix * $h_{t-1}$

Code Sample for $f_t$:

import numpy as np
import math

def sigmoid(values):
    sigmoid_applied = []
    for value in values:
        result = 1 / (1 + math.pow(math.e, -value))
        sigmoid_applied.append(result)
    return np.array(sigmoid_applied)

w1 = np.random.uniform(0, 1, size=[hidden_vector_len, input_len])
w2 = np.random.uniform(0, 1, size=[hidden_vector_len, hidden_vector_len])

f_t = sigmoid(np.dot(w1, input) + np.dot(w2, prev_hidden_state)) # Its matrix multiplication and not just simple multiplication

Note - There is also a bias term which I haven't included here for simplicity

If you understood $f_t$, you can do the same for other states also.

If you feel I am wrong anywhere in this post, then please do consider adding a comment