The idea is simple, but it requires some time to develop.
Assumption: I am assuming in your problem the final model will have seen all possible shapes.
What your algorithm needs is a convolutional NN to understand each shape by extracting features, but you just need to be very careful with pooling.
Then what you need is a recurrent NN. In the example you ...