I have the following program for my neural network:

n_steps = 9
n_inputs = 36
n_neurons = 50
n_outputs = 1
n_layers = 2
learning_rate = 0.0001
batch_size =100
n_epochs = 1000#200 
train_set_size = 1000
test_set_size = 1000
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs],name="input")
y = tf.placeholder(tf.float32, [None, n_outputs],name="output")
layers = [tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer))
         for layer in range(n_layers)]    layers.append(tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer)))
multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, n_neurons]) 
stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)
outputs = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])
outputs = outputs[:,n_steps-1,:]

I want to know whether my network is fully connected or not?
When I try to see the variables, I see:


The output is:

[<tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/kernel:0' shape=(86, 200) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/bias:0' shape=(200,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_f_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_i_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_o_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/kernel:0' shape=(100, 200) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/bias:0' shape=(200,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_f_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_i_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_o_diag:0' shape=(50,) dtype=float32_ref>]

I didn't understood whether each layer is getting the complete inputs or not.
I want to know whether the following figure is correct for the above code:
enter image description here

If this is not then what is the figure for the network? Please let me know.

  • 1
    $\begingroup$ Yes, it is a feed forward fully connected network. $\endgroup$ – Patel Sunil Jan 28 '19 at 11:07
  • $\begingroup$ @PatelSunil Can you please tell me why there is 86,200. Why 200? Why 86? I know 50 is the neurons per layer. Then why it starts with 100, 200? Why 100? I am a bit confused here. Please help me know. $\endgroup$ – Jaffer Wilson Jan 28 '19 at 11:21
  1. Is figure = code?

    No. Your figure shows a fully connected feed forward network (MLP). But in your code you are using a two layer LSTM with peepholes. For the visualization of LSTMs, blocks are usually used for each layer.

    Here is a figure of the LSTM with peepholes which is the base of the tensorflow implementation (Source: Paper, fig. 1).

enter image description here

  1. Why size 86?

    The input is concatenated with the hidden state: n_inputs + n_neurons = 36 + 50 = 86.

  2. Why size 100 in the second layer?

    The second LSTM layer gets input of size 50 by the first layer (n_neurons) which is concatenated with the hidden state of the second layer (of size n_neurons = 50). Therefore you get 50 + 50 = 100.

  3. Why size 200?

    There are four weight matrices of size $86 \times 50$ (fig.: colored circles and the g circle), which seem to be combined to one matrix ($4 \cdot 50$) of size $86 \times 200$ (layer0/kernel).

  4. Why size 50?

    The three variables w_f_diag, w_i_diag and w_o_diag are for the peephole connections (fig: dashed lines) and they have the size of n_neurons = 50.

  • $\begingroup$ Just one more query. Is the model I am using back propagation and forward propagation? or is it just forward propagation? $\endgroup$ – Jaffer Wilson Jan 29 '19 at 8:11
  • 1
    $\begingroup$ The model uses backpropagation for weight adaption during the training process. It (LSTM) is not a feed forward neural network, but a recurrent neural network, as it has recurrent connections (blue arrows in the figure). $\endgroup$ – dexteritas Jan 29 '19 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.