# How to use RNN With Attention Mechanism on Non Textual Data?

Recurrent Neural Networks (RNN) With Attention Mechanism is generally used for Machine Translation and Natural Language Processing. In Python, implementation of RNN With Attention Mechanism is abundant in Machine Translation (For Eg. https://talbaumel.github.io/blog/attention/, however what I would like to do is to use RNN With Attention Mechanism on a temporal data file (not any textual/sentence based data). I have a CSV file with of dimensions 21000 x 1936, which I have converted to a Dataframe using Pandas. The first column is of Datetime Format and last column consists of target classes like "Class1", "Class2", "Class3" etc. which I would like to identify. So in total, there are 21000 rows (instances of data in 10 minutes time-steps) and 1935 features. The last (1936th column) is the label column.

It is predominant from existing literature that an Attention Mechanism works quite well when coupled into the RNN. I am unable to locate any such implementation of RNN with Attention Mechanism, which can also provide a visualisation as well. Any help in this regard would be highly appreciated. Cheers!

Project Definition

• Labelled data set contains 21 K rows; 1,936 features; and 1 textual label
• Label can be 1 of 14 possible categories
• The first feature is a time stamp reflecting exact or approximate 10 minute sampling period
• Data content not primarily natural language
• The intention is to learn the function mapping the features to the label
• Visualization to observe training intermediate and final results
• Hoping to simplify implementation using already implemented algorithms and development support

Use of Recurrent Artificial Network Learning

It is correct that recurrent networks are designed for temporally related data. The later variants of the original RNN design are most apt to produce favorable results. One of the most effective of these variants is the GRU network cell, which is well represented in all the main machine learning libraries, and visualization hooks in those libraries are well documented.

Various Meanings of Attention Mechanisms

The belief that an attention mechanism beyond those built into the RNN design are needed to emphasize important features may be over-complicating the problem. The parameters of the GRU and the other RNN variants already focus attention on particular features during learning. Even a basic feed forward network does that, but the MLP (multilayer perceptron) does not recognize feature trends temporally, so the use of RNN variants is smart.

There are other kinds of attention mechanisms that are not inside each cell of a network layer. Research into advanced attention based designs that involve oversight, various forms of feedback from the environment, recursion, or generative designs is ongoing. As the question indicates, those are targeted for natural language work. There is also attention based design for motion and facial recognition and automated walking, driving, and piloting systems. They are designed, tested, optimized, and evolving for the purpose of natural language processing or robotics, not 1,936 feature rows. It is unlikely that those systems can be morphed into something any more effective than a GRU network for this project without considerable further R&D.

Output Layer and Class Encoding

The 14 labels should be coded as 14 of the 16 permutations of a 4 bit output prior to training. And the loss function should dissuade the two illegal permutations.

[Of the] 1936 features, one of them [is] date-time timestamps and [the] rest [are] numeric. ... Can you please suggest the format of the input? Should I convert each column of feature to a list and create a list of lists or some other way around?

Regardless of what types the library you use expect as inputs, the theory is clear. Features with a finite set of fixed discrete values are ordinals. The magnitude of their information content is given in bits $$b$$ as follows, where $$p$$ is the total number of possible discrete values for the feature.

$$b = \log_2 p$$

This is also true of the timestamp, which has a specific possible range and time resolution, where $$t_{\emptyset}$$ is the initial timestamp where the project or its data began and $$t_{res}$$ is the time of one resolution step.

$$b_{timestamp} = \log_2 \frac {t_{max} - t_\emptyset} {t_{res}}$$

The label also has a range. If the range is a fixed set of permutations, then assign an integer to each, starting with zero, to encode them. If the range of the text is unknown, use a library or utility that converts words or phrases to numbers. One popular one is word2vec.

Integrating the features to reduce the number of input bits actually wastes a layer, so don't do that. The total information is given as this.

$$b_{total} = \sum_{i = 1}^{1,936} b_i$$

The features, if they are real numbers, can remain so. The input layer of an artificial network expects a number entering the data flow for each cell. One can change the data type of the numbers to reduce computational complexity if no overflow or other mapping issue will occur. This is where the above information content can be useful in understanding how far one can go in collapsing the information into a smaller computational footprint.

• Thanks for the awesome reply. I understand that I need to encode the outputs based on 4 bit combinations as you rightly mention. But I am quite confused as to what form the input would take. My 1936 features (with one of them being datetime timestamps) and rest all numeric features need to be used for this multiclass classification of the 14 classes of output. But to use a GRU (or for that matter any simple RNN), can you please suggest the format of the input, I mean should I convert each column of feature to a list and create a list of lists or some other way around? Jan 20 '19 at 23:54

The article referenced doesn't highlight the most common terms, which is why you may not be finding what you want. We've found that not all the designs have public implementations in every language. Some we've had to translate from Python or Java to C++ to get the speed we needed for embedded applications. Attention mechanisms are not one thing either. There are attention mechanisms in the cells, before the layers, after the layers, as oversight mechanisms, and in push-pull balancing arrangements. The common cell using attention ideas use gates in the cell and the article only briefly mentions it. That's the long-short term memory cell, which is a type of RNN. Look for LSTM and B-LSTM for the first examples, which are gated. The GRU is also a type of RNN cell with gating. This paper is recent and proposes two mechanisms: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/16126/16099. If you search for DiSAN there are a few implementations. The newest kind, which has promise is not really an RNN strictly. That's the recursive network, which can have attention mechanisms but can build learned graph structures instead of tensors. That may also be what you want.

• Thanks a lot for the reply. I went through the documentation and this Paper. I would explain you what I really want to accomplish. My data is of high dimensions (21392X1972). The 1972th column has labels (14 labels, thereby multi-class) which I would like to predict based on remaining features. I want to use RNN for this purpose but with an attention mechanism, so that the important features are emphasised on by the Recurrent Neural Net. I am unable to find any such method to accomplish this. Can you suggest a suitable way/direction to move forward in this regard? Cheers. Jan 17 '19 at 13:12
• @JoyjitChatterjee, I'd need to understand what the data represents. The only thing I can add from what you added in the comment is that you need 4 bits output and the error function must abhor the two 4-bit combinations that are invalid. (2^4 - 14 = two). Can you share more about whether the data is a time series or whether they are independent examples or independent examples of time series? You indicated it is not textual. Is it materials science data or some other scientific data? Account transaction records? Images? Aggregated statistics? Jan 19 '19 at 4:09
• Thanks a lot for the reply again. It is a data related to wind turbines where the first column has timestamps (datetime datatype at generally 10 minutes interval, except when there are some faults etc.). All features except first column are numeric values like meteorological data , Electrical data, mechanical data, control system data and so on. And 14 classes are string type, representing 13 different types of fault labels, and a No Fault Class (total 14 string type labels thereby). Jan 20 '19 at 23:45
• @JoyjitChatterjee, Is this part of a control loop or just for reporting? Jan 21 '19 at 4:13
• It is just the data recorded from various sensors... And has been labelled which I would like to classify based on multi class classification but using RNNs and am not sure how to format the data in this regard. Jan 21 '19 at 7:42