Is my Neural Network program fully connected?

Question

I have the following program for my neural network:

n_steps = 9
n_inputs = 36
n_neurons = 50
n_outputs = 1
n_layers = 2
learning_rate = 0.0001
batch_size =100
n_epochs = 1000#200 
train_set_size = 1000
test_set_size = 1000
tf.reset_default_graph()
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs],name="input")
y = tf.placeholder(tf.float32, [None, n_outputs],name="output")
layers = [tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer))
         for layer in range(n_layers)]    layers.append(tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer)))
multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, n_neurons]) 
stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)
outputs = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])
outputs = outputs[:,n_steps-1,:]

I want to know whether my network is fully connected or not?
When I try to see the variables, I see:

multi_layer_cell.weights

The output is:

[<tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/kernel:0' shape=(86, 200) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/bias:0' shape=(200,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_f_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_i_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_o_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/kernel:0' shape=(100, 200) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/bias:0' shape=(200,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_f_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_i_diag:0' shape=(50,) dtype=float32_ref>,
 <tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_o_diag:0' shape=(50,) dtype=float32_ref>]

I didn't understood whether each layer is getting the complete inputs or not.
I want to know whether the following figure is correct for the above code:

If this is not then what is the figure for the network? Please let me know.

Answer 1

1. Is figure = code?

   No. Your figure shows a fully connected feed forward network (MLP). But in your code you are using a two layer LSTM with peepholes. For the visualization of LSTMs, blocks are usually used for each layer.

   Here is a figure of the LSTM with peepholes which is the base of the tensorflow implementation (Source: Paper, fig. 1).

2. Why size 86?

   The input is concatenated with the hidden state: n_inputs + n_neurons = 36 + 50 = 86.

3. Why size 100 in the second layer?

   The second LSTM layer gets input of size 50 by the first layer (n_neurons) which is concatenated with the hidden state of the second layer (of size n_neurons = 50). Therefore you get 50 + 50 = 100.

4. Why size 200?

   There are four weight matrices of size $86 \times 50$ (fig.: colored circles and the g circle), which seem to be combined to one matrix ($4 \cdot 50$) of size $86 \times 200$ (layer0/kernel).

5. Why size 50?

   The three variables w_f_diag, w_i_diag and w_o_diag are for the peephole connections (fig: dashed lines) and they have the size of n_neurons = 50.