0
$\begingroup$

Usually for DNN, I have the training data of matching X (2D) to Y (2D), for example, XOR data:

X = [[0,0],[0,1],[1,0],[1,1]];
Y = [[0],  [1],  [1],  [0]  ];

However, RNN seems strange, I don't get it how to match X to Y, input of RNN layer is 3D and output is 2D (rightclick to open in new tab): https://colab.research.google.com/drive/17IgFuxOYgN5fNO9LKwDijEBkIeWNPas6

import tensorflow as tf;

x = [[[1],[2],[3]], [[4],[5],[6]]];
bsize = 2;
times = 3;

#3d input
input = tf.placeholder(tf.float32, [bsize,times,1]);

cell  = tf.keras.layers.LSTMCell(20);
rnn   = tf.keras.layers.RNN(cell);
hid   = rnn(input);

sess = tf.Session();
init = tf.global_variables_initializer();
sess.run(init);

#results in 2d
print(sess.run(hid, {input:x}));

The example data seen on https://www.tensorflow.org/tutorials/sequences/recurrent are:

 t=0  t=1    t=2  t=3     t=4
[the, brown, fox, is,     quick]
[the, red,   fox, jumped, high]

How to map these data from X (3D input for RNN layer) to Y (2D)? (Y is 2D because RNN layer output is 2D).

$\endgroup$
  • 1
    $\begingroup$ I'm not 100% sure I underststood the question but I'll try to answer it. By default keras is returning a tensor of shape: (batch_size, state_size) which corresponds to the last hidden step for each batch. If you want to return all the hidden states you need to set return_sequences to True, and it will return a 3d tensor with shape: (batch_size, timesteps, output_size). $\endgroup$ – razvanc92 Sep 18 at 11:21
  • 1
    $\begingroup$ @razvanc92 actually 2d or 3d is the side matter, the main thing i would like to ask is how to create the training data X, Y for the brown-fox-red-fox example data on tensorflow.org/tutorials/sequences/recurrent $\endgroup$ – datdinhquoc Sep 18 at 11:54
  • $\begingroup$ The very first step you need to do, is to embed each work into a fixed size vector. Afterwards from my understanding, at time t=0 the input will be X[:, 0] or [the, the] in your case. The target will be the next time stamp t=1 X[:,1] or [brown, red]. Following this logic you got X already, and for Y it will be X[:, 1:] (all the words from X, pushed by one plus probably a special character to mark the end of the sentence.). $\endgroup$ – razvanc92 Sep 18 at 12:15
  • $\begingroup$ @razvanc92 i know about embedding, but i'm just trying out with class index first $\endgroup$ – datdinhquoc Sep 19 at 2:25
0
$\begingroup$

I found out how to get 3D output from LSTMCell so that I can matmul with output weights + biases and subtract with expected values:

  • Inputs & expecteds should be: placeholder(,[times,batch_size,num_inp]) instead of batch_size first then times. However, tf.keras.layers.LSTM will ask for [batch_size,times,num_inp]
  • Use tf.nn.static_rnn with a list of inputs, instead of 1 input

Source code:

import tensorflow as tf;

x = [[[1],[2],[3]],[[4],[5],[6]]];
times = 2;
bsize = 3;

#3d input
inputs = tf.placeholder(tf.float32, [times,bsize,1]);

cell   = tf.nn.rnn_cell.BasicRNNCell(20);
hids,_ = tf.nn.static_rnn(cell,tf.unstack(inputs,times),dtype=tf.float32);

sess = tf.Session();
init = tf.global_variables_initializer();
sess.run(init);

#results in 2d
print(sess.run(hids, {inputs:x}));
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.