Inner working of Bidirectional RNNs

Question

I'm trying to understand how Bidirectional RNNs work.

Specifically, I want to know whether a single cell is used with different states, or two different cells are used, each having independent parameters.

In pythonic pseudocode,

Implementation 1:

cell = rev_cell = RNNCell()
cell_state = cell.get_initial_state()
rev_cell_state = rev_cell.get_initial_state()
for i in range(len(series)):
    output, cell_state = cell(series[i], cell_state)
    rev_output, rev_cell_state = rev_cell(series[-i-1], rev_cell_state)
    final_output = concatenate([output, rev_output])

Implementation 2:

cell = RNNCell()
rev_cell = RNNCell()
cell_state = cell.get_initial_state()
rev_cell_state = rev_cell.get_initial_state()
for i in range(len(series)):
    output, cell_state = cell(series[i], cell_state)
    rev_output, rev_cell_state = rev_cell(series[-i-1], rev_cell_state)
    final_output = concatenate([output, rev_output])

Which of the above implementations is correct? Or is the working of Bidirectional RNNs completely different altogether?

Answer 1

The second implementation looks more correct and inline with how Bidirectional is defined. Specifically, bidirectionality doen't change the forward/backward logic of either direction, and just merges (concat/sum/...) the outputs of forward/backward at a matching timestep t.

You can check how Keras implements it here. There are distinct self.forward_layer and self.backward_layer that are initialized separately.

Your for loop doesn't look ok though. To calculate the output at time step 0 you'd have to calculate forward(0) and backward(n), which means you have to run backward on all the samples first. In practice, each direction is calculated separately and the results are merged afterwards. Check the implementaion in Keras here.