0
$\begingroup$

I am watching the Sequence models course taught by Andrew Ng. I am a bit lost on the SimpleRNN lecture. As per the lecture, at each time step, there's an output y from a hidden layer and an input activation from the preceding layer. However, I do not understand what's the difference between the activation and output $y$ at any time step (except the first).

As per my understanding -

$$y = activation(w*x+b)$$

Therefore $y_t$ and $activation_t$ should be the same. I have included a figure to describe my issue.

enter image description here

In the picture, what's the difference between $\hat{a}^{\langle t \rangle}$ and $\hat{y}^{\langle t \rangle}$?

Here's the link of the lesson - https://www.coursera.org/learn/nlp-sequence-models/lecture/ftkzt/recurrent-neural-network-model

$\endgroup$
0

1 Answer 1

1
$\begingroup$

enter image description here

They are not the same always, the activation (as you called it) here is the hidden state (I think it's a in your slide), now you can have the output is the same as the hidden state (for example that may represent a word embedding) but for most problems you need to do more operations on the hidden state to calculate the output.

We do that because what you need as output in the current step may not be the whole hidden state (which may contatin info that required in future steps), also you may need to project the hidden state, for example if you want to predict a word you need to project the hidden state into a vector in $\mathbb{R}^{|V|}$ where $V$ is your vocabulary, and then apply a softmax over it (like in the equations above).

$\endgroup$
2
  • $\begingroup$ So, if I understand correctly, the weight wya(in the image) or V(in eq 10.10) can be considered analogous to inserting a dense layer after the RNN layer. $\endgroup$ Commented Jul 1, 2022 at 1:04
  • 1
    $\begingroup$ @desert_ranger Yes. $\endgroup$
    – Kais Hasan
    Commented Jul 1, 2022 at 9:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .