Recurrent Neural Networks (RNN) With Attention Mechanism is generally used for Machine Translation and Natural Language Processing. In Python, implementation of RNN With Attention Mechanism is abundant in Machine Translation (For Eg. https://talbaumel.github.io/blog/attention/, however what I would like to do is to use RNN With Attention Mechanism on a temporal data file (not any textual/sentence based data). I have a CSV file with of dimensions 21000 x 1936, which I have converted to a Dataframe using Pandas. The first column is of Datetime Format and last column consists of target classes like "Class1", "Class2", "Class3" etc. which I would like to identify. So in total, there are 21000 rows (instances of data in 10 minutes time-steps) and 1935 features. The last (1936th column) is the label column.

It is predominant from existing literature that an Attention Mechanism works quite well when coupled into the RNN. I am unable to locate any such implementation of RNN with Attention Mechanism, which can also provide a visualisation as well. Any help in this regard would be highly appreciated. Cheers!


1 Answer 1


Project Definition

  • Labelled data set contains 21 K rows; 1,936 features; and 1 textual label
  • Label can be 1 of 14 possible categories
  • The first feature is a time stamp reflecting exact or approximate 10 minute sampling period
  • Data content not primarily natural language
  • The intention is to learn the function mapping the features to the label
  • Visualization to observe training intermediate and final results
  • Hoping to simplify implementation using already implemented algorithms and development support

Use of Recurrent Artificial Network Learning

It is correct that recurrent networks are designed for temporally related data. The later variants of the original RNN design are most apt to produce favorable results. One of the most effective of these variants is the GRU network cell, which is well represented in all the main machine learning libraries, and visualization hooks in those libraries are well documented.

Various Meanings of Attention Mechanisms

The belief that an attention mechanism beyond those built into the RNN design are needed to emphasize important features may be over-complicating the problem. The parameters of the GRU and the other RNN variants already focus attention on particular features during learning. Even a basic feed forward network does that, but the MLP (multilayer perceptron) does not recognize feature trends temporally, so the use of RNN variants is smart.

There are other kinds of attention mechanisms that are not inside each cell of a network layer. Research into advanced attention based designs that involve oversight, various forms of feedback from the environment, recursion, or generative designs is ongoing. As the question indicates, those are targeted for natural language work. There is also attention based design for motion and facial recognition and automated walking, driving, and piloting systems. They are designed, tested, optimized, and evolving for the purpose of natural language processing or robotics, not 1,936 feature rows. It is unlikely that those systems can be morphed into something any more effective than a GRU network for this project without considerable further R&D.

Output Layer and Class Encoding

The 14 labels should be coded as 14 of the 16 permutations of a 4 bit output prior to training. And the loss function should dissuade the two illegal permutations.

Response to Comments

[Of the] 1936 features, one of them [is] date-time timestamps and [the] rest [are] numeric. ... Can you please suggest the format of the input? Should I convert each column of feature to a list and create a list of lists or some other way around?

Regardless of what types the library you use expect as inputs, the theory is clear. Features with a finite set of fixed discrete values are ordinals. The magnitude of their information content is given in bits $b$ as follows, where $p$ is the total number of possible discrete values for the feature.

$$ b = \log_2 p $$

This is also true of the timestamp, which has a specific possible range and time resolution, where $t_{\emptyset}$ is the initial timestamp where the project or its data began and $t_{res}$ is the time of one resolution step.

$$ b_{timestamp} = \log_2 \frac {t_{max} - t_\emptyset} {t_{res}} $$

The label also has a range. If the range is a fixed set of permutations, then assign an integer to each, starting with zero, to encode them. If the range of the text is unknown, use a library or utility that converts words or phrases to numbers. One popular one is word2vec.

Integrating the features to reduce the number of input bits actually wastes a layer, so don't do that. The total information is given as this.

$$ b_{total} = \sum_{i = 1}^{1,936} b_i $$

The features, if they are real numbers, can remain so. The input layer of an artificial network expects a number entering the data flow for each cell. One can change the data type of the numbers to reduce computational complexity if no overflow or other mapping issue will occur. This is where the above information content can be useful in understanding how far one can go in collapsing the information into a smaller computational footprint.

  • $\begingroup$ Thanks for the awesome reply. I understand that I need to encode the outputs based on 4 bit combinations as you rightly mention. But I am quite confused as to what form the input would take. My 1936 features (with one of them being datetime timestamps) and rest all numeric features need to be used for this multiclass classification of the 14 classes of output. But to use a GRU (or for that matter any simple RNN), can you please suggest the format of the input, I mean should I convert each column of feature to a list and create a list of lists or some other way around? $\endgroup$
    – JChat
    Jan 20, 2019 at 23:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .