1
$\begingroup$

I want an LSTM to output one of two classes (Y, N), per frame, based on all the input so far.
My original inputs are very long (~100000 samples long, far more than a standard LSTM training can handle due to vanishing gradients).

  1. If the last seen instance out of the tokens (A, B) was A, output Y.
  2. If the last seen instance out of the tokens (A, B) was B, output N.
  3. The very long sequence is guaranteed to start with either A or B.

If the sequence was short, this would be quite easy.

For example, the following top lines and bottom lines correspond to inputs and required outputs:

ABCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
YNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN

ACCCCCCCCCBCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCCACCCCCCCC
YYYYYYYYYYNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNYYYYYYYYYYYYYYYYYYYYYYYYYY

Looks easy enough, just push batches comprised of chunks of the long sequence to the LSTM and have a coffee, right?
However, for my case, the available inputs are (A, B, C), of which (A, B) are extremely rare, meaning I can have batches comprised of 100% C's. The LSTM has no chance then, if not fed with some current state, telling it about the last A or B seen.
Unfortunately, this "state" is really something learned, and I can't just feed it as input AFAIK.

CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
????????????????????????????????????????????????????????????????????

I am looking for a standard practice, or other references on how to train an LSTM or other RNN based model to be able to classify based on rare events far in history.

I hope this is clear, if not please ask and I will edit.

Please note that the data is labeled, and labeling can't be generated automatically for this task. The above is just an example for ease of understanding, the reality is more complicated.

Improve this question
$\endgroup$
5
  • $\begingroup$ Why do you need an LSTM here? Is either A or B a composite event over multiple time steps? If not, then what is wrong with training a classifier for A, B, C and applying your simple logic for Y/N over a basic variable? There may still be an issue due to unbalanced cases for A/B/C, but it seems more tractable than throwing an LSTM at the problem if you don't actually care about anything subtle over many timestamps (i.e. the Cs are all uninteresting and do not affect eventual classification of A or B), just the classification of individual events. $\endgroup$
    – Neil Slater
    Apr 13, 2021 at 15:57
  • $\begingroup$ @NeilSlater The events are indeed composite. But even if they weren't, I am not following how you propose not using some sort of a model with memory. By feeding back that classification variable into the network? Isn't that LSTM with less steps [which may be ok, just making sure]? $\endgroup$
    – Gulzar
    Apr 13, 2021 at 16:03
  • $\begingroup$ More than the A/B events are composite, the logic for closing the loop between Y/N to A/B is composite. For example, the switch may not be immediate as stated in the question. $\endgroup$
    – Gulzar
    Apr 13, 2021 at 16:05
  • $\begingroup$ My point is that your "model with a memory" can literally be some Python (or whatever) var last_seen = 'A'. Unless there is some subtlety and some Cs are different from other Cs (in terms of whether or not A or B eventually happens), then forcing the problem to be "I must write an LSTM with very long memory" might be making a rod for your own back. It is hard to tell because you have simplified things, which is fine. However, i don't see that the Y/N switch is "composite" - the rules you state are simple and absolute, and there is no statement in any other direction. $\endgroup$
    – Neil Slater
    Apr 13, 2021 at 17:00
  • $\begingroup$ @Neil Slater alright, this is fair. I will haveto come up with something a bit more elaborated then. I will sleep on it. $\endgroup$
    – Gulzar
    Apr 13, 2021 at 21:45

1 Answer 1

Reset to default
1
$\begingroup$

How about a Temporal Convolutional Network? It feels like for such a long sequences having the recurrent/memory based approach is not too feasible. But, intuitively, the 1D convolutions should be able to pick out those rare features from your extremely long sequences.

There are also claims that TCNs are comparable to RNNs in performance on common tasks, so there's that.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .