2
$\begingroup$

I have a large 2D grid having 30k rows and 35k columns, so a total of 30x35k grid cells. Each grid cell is represented by a unique integer number (identity of grid cell). I have several trajectories that passes through these grid cells. Each trajectory is represented by a sequence of numbers (that are grid cells through which the trajectory passes through).

I want to solve the problem of trajectory prediction by giving the partial trajectory as input and predict the full trajectory. This becomes a sequence to sequence problem, where all sequences are integer values by default.

I am trying to solve this problem through encoder-decoder LSTM architecture. Most tutorials/examples regarding sequence to sequence on net are on machine translations in which vocabularies or characters are one-hot-encoded to represent the text as integer values. When I hot-encode my sequence values the one-hot vector becomes very large because there are (30x35)k grid cells, the program has given memory overflow error (because each vector has of size 1 million).

I am confused here, do I need to treat grid identity as categorical variable? because all grid identities are numeric numbers but these identities are not comparable (like prices).

Do I need to hot-encode my integer values in my sequence? Or is there any other alternative to solve this problem? I also appreciate if you suggest me the similar tutorials having the sequence to sequence prediction problem.

$\endgroup$
0
$\begingroup$

One hot encoding helps a lot. You can actually one hot encode each batch just before the training to reduce memory usage. One hot encoding makes the input data more intuitive for the network as a numeric value requires the network to do multiple comparisons to understand the value. For examples please tell me which deep learning framework you are using so I can give you some links or code. Hope I can help you.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I am trying to solve using encoder-decoder LSTM. especially I also need the intuition/concept of onehot batch encoding, how it works? $\endgroup$ – Asif Khan Nov 15 '19 at 13:09
  • $\begingroup$ What do you mean by one hot batch encoding? Thanks @Asif Khan $\endgroup$ – Clement Hui Nov 15 '19 at 13:20
  • $\begingroup$ The statement in your answer "You can actually one hot encode each batch just before the training to reduce memory usage.". how to do this $\endgroup$ – Asif Khan Nov 15 '19 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.