# Is my Neural Network program fully connected?

I have the following program for my neural network:

n_steps = 9
n_inputs = 36
n_neurons = 50
n_outputs = 1
n_layers = 2
learning_rate = 0.0001
batch_size =100
n_epochs = 1000#200
train_set_size = 1000
test_set_size = 1000
tf.reset_default_graph()
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs],name="input")
y = tf.placeholder(tf.float32, [None, n_outputs],name="output")
layers = [tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer))
for layer in range(n_layers)]    layers.append(tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer)))
multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, n_neurons])
stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)
outputs = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])
outputs = outputs[:,n_steps-1,:]


I want to know whether my network is fully connected or not?
When I try to see the variables, I see:

multi_layer_cell.weights


The output is:

[<tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/kernel:0' shape=(86, 200) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/bias:0' shape=(200,) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_f_diag:0' shape=(50,) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_i_diag:0' shape=(50,) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_o_diag:0' shape=(50,) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/kernel:0' shape=(100, 200) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/bias:0' shape=(200,) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_f_diag:0' shape=(50,) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_i_diag:0' shape=(50,) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_o_diag:0' shape=(50,) dtype=float32_ref>]


I didn't understood whether each layer is getting the complete inputs or not.
I want to know whether the following figure is correct for the above code:

If this is not then what is the figure for the network? Please let me know.

• Yes, it is a feed forward fully connected network. – Patel Sunil Jan 28 at 11:07
• @PatelSunil Can you please tell me why there is 86,200. Why 200? Why 86? I know 50 is the neurons per layer. Then why it starts with 100, 200? Why 100? I am a bit confused here. Please help me know. – Jaffer Wilson Jan 28 at 11:21

1. Is figure = code?

No. Your figure shows a fully connected feed forward network (MLP). But in your code you are using a two layer LSTM with peepholes. For the visualization of LSTMs, blocks are usually used for each layer.

Here is a figure of the LSTM with peepholes which is the base of the tensorflow implementation (Source: Paper, fig. 1).

1. Why size 86?

The input is concatenated with the hidden state: n_inputs + n_neurons = 36 + 50 = 86.

2. Why size 100 in the second layer?

The second LSTM layer gets input of size 50 by the first layer (n_neurons) which is concatenated with the hidden state of the second layer (of size n_neurons = 50). Therefore you get 50 + 50 = 100.

3. Why size 200?

There are four weight matrices of size $$86 \times 50$$ (fig.: colored circles and the g circle), which seem to be combined to one matrix ($$4 \cdot 50$$) of size $$86 \times 200$$ (layer0/kernel).

4. Why size 50?

The three variables w_f_diag, w_i_diag and w_o_diag are for the peephole connections (fig: dashed lines) and they have the size of n_neurons = 50.

• Perfecto amigo.. this is what I was willing to here.... thank you. – Jaffer Wilson Jan 29 at 5:54
• Just one more query. Is the model I am using back propagation and forward propagation? or is it just forward propagation? – Jaffer Wilson Jan 29 at 8:11
• The model uses backpropagation for weight adaption during the training process. It (LSTM) is not a feed forward neural network, but a recurrent neural network, as it has recurrent connections (blue arrows in the figure). – dexteritas Jan 29 at 8:27