2
$\begingroup$

I want to code up one time step in a LSTM. My focus is on understanding the functioning of the forget gate layer, input gate layer, candidate values, present and future cell states.

Lets assume that my hidden state at t-1 and xt are the following. For simplicity, lets assume that the weight matrices are identity matrices, and all biases are zero.

htminus1 = np.array( [0, 0.5, 0.1, 0.2, 0.6] )
xt = np.array( [-0.1, 0.3, 0.1, -0.25, 0.1] )

I understand that forget state is sigmoid of htminus1 and xt

So, is it?

ft = 1 / ( 1 + np.exp( -( htminus1 + xt ) ) )

>> ft = array([0.47502081, 0.68997448, 0.549834  , 0.4875026 , 0.66818777])

I am referring to this link to implement of one iteration of one block LSTM. The link says that ft should be 0 or 1. Am I missing something here?

How do I get the forget gate layer as per schema given in the below mentioned picture? An example will be illustrative for me.

enter image description here

Along the same lines, how do I get the input gate layer, it and vector of new candidate values, \tilde{C}_t as per the following picture?

enter image description here

Finally, how do I get the new hidden state ht as per the scheme given in the following picture?

A simple, example will be helpful for me in understanding. Thanks in advance.

enter image description here

$\endgroup$
1
$\begingroup$

enter image description here

This is an image to better understand lstm... At $f_t$, we are taking the sigmoid of a weight matrix * the input at the current timestep + another weight matrix * $h_{t-1}$

Code Sample for $f_t$:

import numpy as np
import math

def sigmoid(values):
    sigmoid_applied = []
    for value in values:
        result = 1 / (1 + math.pow(math.e, -value))
        sigmoid_applied.append(result)
    return np.array(sigmoid_applied)

w1 = np.random.uniform(0, 1, size=[hidden_vector_len, input_len])
w2 = np.random.uniform(0, 1, size=[hidden_vector_len, hidden_vector_len])

f_t = sigmoid(np.dot(w1, input) + np.dot(w2, prev_hidden_state)) # Its matrix multiplication and not just simple multiplication

Note - There is also a bias term which I haven't included here for simplicity

If you understood $f_t$, you can do the same for other states also.

If you feel I am wrong anywhere in this post, then please do consider adding a comment

$\endgroup$
2
  • $\begingroup$ Shouldn't the size of w2 be [hidden_vector_len, hidden_vector_len] ? $\endgroup$ – razvanc92 Dec 10 '20 at 9:07
  • $\begingroup$ @razvanc92 Fixed! Thanks for Pointing Out! $\endgroup$ – NITIN AGARWAL Dec 10 '20 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.