1
$\begingroup$

enter image description here

In the above diagram, the shape of some of the matrices can be seen in the yellow highlight. For instance:

The hidden state at timestep t-1 ($h_{t-1}$) has shape $(na, m)$

The input data at timestep t ($x_{t}$) has shape $(nx, m)$

$Z_{t}$ has shape $(na+nx, m)$ since the hidden state and input data are concatenated in LSTMs.

$W_{c}$ has shape $(na, na+nx)$

$W_{c}$$Z_{t}$ has shape $(na, m)$ = $i_{t}$

$W_{i}$$Z_{t}$ has shape $(na, m)$ = $ĉ_{t}$

When working through the network to the point $i_{t}$ and $ĉ_{t}$, how can these two be dot producted when the multiplication is not of the form (m x n)(n x p) as per the matrix multiplication definition?:

enter image description here

$\endgroup$
1
$\begingroup$

Turns out the reason is because, for places where a dot is shown in the image above, they're actually element-wise multiplications, not dot products. A lot of sources use an X or . to denote multiplication, but don't clearly indicate they mean element-wise multiplication.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.