2
$\begingroup$

I have a large 2D grid having 30k rows and 35k columns, so a total of 30x35k grid cells. Each grid cell is represented by a unique integer number (identity of grid cell). I have several trajectories that passes through these grid cells. Each trajectory is represented by a sequence of numbers (that are grid cells through which the trajectory passes through).

I want to solve the problem of trajectory prediction by giving the partial trajectory as input and predict the full trajectory. This becomes a sequence to sequence problem, where all sequences are integer values by default.

I am trying to solve this problem through encoder-decoder LSTM architecture. Most tutorials/examples regarding sequence to sequence on net are on machine translations in which vocabularies or characters are one-hot-encoded to represent the text as integer values. When I hot-encode my sequence values the one-hot vector becomes very large because there are (30x35)k grid cells, the program has given memory overflow error (because each vector has of size 1 million).

I am confused here, do I need to treat grid identity as categorical variable? because all grid identities are numeric numbers but these identities are not comparable (like prices).

Do I need to hot-encode my integer values in my sequence? Or is there any other alternative to solve this problem? I also appreciate if you suggest me the similar tutorials having the sequence to sequence prediction problem.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

One hot encoding helps a lot. You can actually one hot encode each batch just before the training to reduce memory usage. One hot encoding makes the input data more intuitive for the network as a numeric value requires the network to do multiple comparisons to understand the value. For examples please tell me which deep learning framework you are using so I can give you some links or code. Hope I can help you.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I am trying to solve using encoder-decoder LSTM. especially I also need the intuition/concept of onehot batch encoding, how it works? $\endgroup$
    – Asif Khan
    Nov 15, 2019 at 13:09
  • $\begingroup$ What do you mean by one hot batch encoding? Thanks @Asif Khan $\endgroup$
    – Clement
    Nov 15, 2019 at 13:20
  • $\begingroup$ The statement in your answer "You can actually one hot encode each batch just before the training to reduce memory usage.". how to do this $\endgroup$
    – Asif Khan
    Nov 15, 2019 at 14:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .