How does the hidden activation differ from the output, at any time step for a SimpleRNN?

Question

I am watching the Sequence models course taught by Andrew Ng. I am a bit lost on the SimpleRNN lecture. As per the lecture, at each time step, there's an output y from a hidden layer and an input activation from the preceding layer. However, I do not understand what's the difference between the activation and output $y$ at any time step (except the first).

As per my understanding -

$$y = activation(w*x+b)$$

Therefore $y_t$ and $activation_t$ should be the same. I have included a figure to describe my issue.

In the picture, what's the difference between $\hat{a}^{\langle t \rangle}$ and $\hat{y}^{\langle t \rangle}$?

Here's the link of the lesson - https://www.coursera.org/learn/nlp-sequence-models/lecture/ftkzt/recurrent-neural-network-model

Answer 1

They are not the same always, the activation (as you called it) here is the hidden state (I think it's a in your slide), now you can have the output is the same as the hidden state (for example that may represent a word embedding) but for most problems you need to do more operations on the hidden state to calculate the output.

We do that because what you need as output in the current step may not be the whole hidden state (which may contatin info that required in future steps), also you may need to project the hidden state, for example if you want to predict a word you need to project the hidden state into a vector in $\mathbb{R}^{|V|}$ where $V$ is your vocabulary, and then apply a softmax over it (like in the equations above).