In the paper What You Get Is What You See: A Visual Markup Decompiler, the authors have proposed a method to extract the features from the CNN and then arrange those extracted features in a grid to pass into an RNN encoder. Here's an illustration.

enter image description here

I can easily extract features from either the existing model, like ResNet, VGG, or make a new CNN model easily as they have described in the paper.

For example, let us suppose, I do this

features = keras.applications.ResNet()(images_array) # just hypothetical

How can I convert these images to the grid?? I am supposed to feed the output of the changed grid to an LSTM Encoder as:

keras.layers.LSTM()(grid) # again, hypothetical type

I just want to know what the author means from changing the output in the grid format.



You must log in to answer this question.