3
$\begingroup$

I have recently begun researching LSTM networks, as I have finished my GA and am looking to progress to something more difficult. I believe I am using the classic LSTM (if that makes any sense) and have a few questions.

Do I need LSTM units everywhere in the network? For example, can I only use LSTM units for the first and last layer and use feedforward units everywhere else?

How do I go about implementing bias values into an LSTM?

Assuming I create a network that predicts the next few words of a sentence, does that mean my outputs should be every possible word that the network could conceivably use?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

For question 1) I dont understand what you are getting at... LSTM cells will work on a contiguous block of inputs, where it sequentially uses the states from a previous time step and the new input to generate the next ones.

question 2) Please look into the LSTM atchitecture LSTM pic As you see, biases are already there, is there somewhere specific you want it, that it isnt?

question 3) Generally yes, but the normalization step can be expensive (such as softmax), so if you want to get clever, you can use negative sampling or hierarchial softmax-- but generally, you you predict a probability over all possible words given the previous text

Improve this answer
$\endgroup$
6
  • $\begingroup$ Okay, I'll try to clarify: 1. A LTSM network (I assume) has more than a single cell. Say I had a 2 inputs a single hidden layer with 3 cells and 2 outputs. Would all those cells be of the LSTM variety? Would their vertical outputs connect at random to the vertical inputs? 2. I see from the diagram you included there are two horizontal line, the memory blocks and the outputs. Ommiting the vertical output and input, would the "horizontal output" feed into it's own "output of previous block"? And would the cells own "memory concurrent block" feed into it's own "memory from previous block" essenti $\endgroup$
    – user11376058
    Commented Jun 5, 2019 at 13:43
  • $\begingroup$ There is only one cell that gets reused every timestep per lstm.... and yes, the output is one of the hidden states of the next layer..... and the memory get fed forward, never backwards $\endgroup$
    – mshlis
    Commented Jun 5, 2019 at 16:21
  • $\begingroup$ I see, so each cell holds its memory independent from one another, only the vertical output gets fed forward. And a network can have multiple lstm cell layers feeding information to an output layer. $\endgroup$
    – user11376058
    Commented Jun 5, 2019 at 16:51
  • $\begingroup$ so there is no weights or dot mult in there? how do you backpropagate that? $\endgroup$
    – user8426627
    Commented Jun 6, 2019 at 0:09
  • $\begingroup$ @user8426627 where ever you see a non-linearity there is a linear operation preceding it... and side note (basic ops like additions and activation fns still have gradients, so youd be able to backprop it regardless, it just wouldnt have learnable params) $\endgroup$
    – mshlis
    Commented Jun 6, 2019 at 12:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .