- custom clustering layers from this tutorial: https://www.dlology.com/blog/how-to-do-unsupervised-clustering-with-keras/
The input for my neural network is based on the code2vec preprocessing where an input source code file (in my case lua file) is first represented as an AST, then all possible paths from leafs to other leafs (through) root are found. These paths are documented as
[Source_node, path, target_node]. In my case the
source_node contains the order of the node in ast (lets say first leaf) and the node type - function, variable, interface, require. Same for
path are all nodes that are traversed to reach
Based on the dataset available to me (18 000 files), I set the number of these triplets (context paths) to be 430. So one file is represented using 430 context paths. If a file has less, it is zero-padded, if it has more, every second context path is chosen until exactly 430 remain. These values are then hashed using
java_hash_code imitation function, and then normalized. The input for the neural network is in the shape of
(, 430, 3) which representes 430 source nodes, 430 paths and 430 target nodes.
So far so good. The tricky part is the purpose of my network was to assign labels to these files, based on if they follow some kind of structure, lets say if the order of the nodes is generally require - variables - functions - interface, or something else entirely. For this purpose I created an autoencoder in Keras with this architecture:
This structure allows me after a training with batch size = 256, epochs = 300, to after 20~ hours train an autoencoder with ~89% accuracy. Then I cut it in the middle to achieve the structure in figure 1 and append a clustering layer. After training the clustering layer I use a custom visualization tool to create AST graphs for each file in said category and I try to find the common attribute for those AST graphs. Now I tried going from very low category numbers (3 categories) to up to 13 categories (label == category), to see if there is some kind of pattern. The common thing I found is that the network categorizes them based on the:
- AST size == number of nodes in AST
- Number of branching
The most concerning thing for me is the categorizing based on AST size which is something I don't want. I tried changing the number of neurons in my last LSTM layer, or adding more LSTM layers so that the resulting representation would be sparser, meaning there would be no "gaps" for files which have a very low number of AST nodes. I hoped that increasing the number of categories (labels) would force the network to predict the category based on something else than the size of AST. The last thing that comes to mind is trying to fiddle with the preprocessing of the files, but this is something I am reluctant to do as the preprocessing is based on the idea of a working concept taken from a scientific paper which yields results.