I have implemented a recurrent neural network in keras with the following architecture: enter image description here

The input for my neural network is based on the code2vec preprocessing where an input source code file (in my case lua file) is first represented as an AST, then all possible paths from leafs to other leafs (through) root are found. These paths are documented as [Source_node, path, target_node]. In my case the source_node contains the order of the node in ast (lets say first leaf) and the node type - function, variable, interface, require. Same for target_node. The path are all nodes that are traversed to reach target_node from source_node.

Based on the dataset available to me (18 000 files), I set the number of these triplets (context paths) to be 430. So one file is represented using 430 context paths. If a file has less, it is zero-padded, if it has more, every second context path is chosen until exactly 430 remain. These values are then hashed using java_hash_code imitation function, and then normalized. The input for the neural network is in the shape of (, 430, 3) which representes 430 source nodes, 430 paths and 430 target nodes.

So far so good. The tricky part is the purpose of my network was to assign labels to these files, based on if they follow some kind of structure, lets say if the order of the nodes is generally require - variables - functions - interface, or something else entirely. For this purpose I created an autoencoder in Keras with this architecture: enter image description here

This structure allows me after a training with batch size = 256, epochs = 300, to after 20~ hours train an autoencoder with ~89% accuracy. Then I cut it in the middle to achieve the structure in figure 1 and append a clustering layer. After training the clustering layer I use a custom visualization tool to create AST graphs for each file in said category and I try to find the common attribute for those AST graphs. Now I tried going from very low category numbers (3 categories) to up to 13 categories (label == category), to see if there is some kind of pattern. The common thing I found is that the network categorizes them based on the:

  • AST size == number of nodes in AST
  • Number of branching

The most concerning thing for me is the categorizing based on AST size which is something I don't want. I tried changing the number of neurons in my last LSTM layer, or adding more LSTM layers so that the resulting representation would be sparser, meaning there would be no "gaps" for files which have a very low number of AST nodes. I hoped that increasing the number of categories (labels) would force the network to predict the category based on something else than the size of AST. The last thing that comes to mind is trying to fiddle with the preprocessing of the files, but this is something I am reluctant to do as the preprocessing is based on the idea of a working concept taken from a scientific paper which yields results.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.