As a researcher, I am getting interested in deep learning (as everyone else:)), and I decided to start with the variational eutoencoders, since I am more interested in unsupervised than supervised learning. I have already read tutorials on the general idea of the mechanism of VAE, and what people usually suggest is to use TensorFlow, Keros, etc. to implement them. However, I do not just want to use machine learning, but also want to develop novel algorithms on top of the existing ones, so I want to deeply understand the mathematical details. And in my experience, in order to understand them, reading is not enough, you need to implement. Any suggestions on where/how to start? I have a general machine learning and programming background, but no experience on deep learning. I even don't know what exactly back-propagation is used for, if this would help.
I even don't know what exactly back-propagation is used for, if this would help.
Programming backprop manually is one of the best exercises in machine learning. In fact, to do this you don't need a special package, like Tensorflor or Keras, just a general computing library like NumPy (and a whiteboard).
Though I dare to say that VAE is not the best thing to implement, if you are not 100% familiar with backprop. It's pretty advanced technique and you risk to mix up fundamental (e.g. backprop) and specific ideas (e.g. deliberately crafted loss function) and slow down your understanding.
I really think that writing a simple feed-forward neural net (including forward and backward passes) is a very good starting point, after which you can move on to VAE details, if you'd like.
Any suggestions on where/how to start?
If you feel confident with algebra and calculus, I highly recommend this Stanford course, especially its assignments. The course itself is about computer vision and convolutional neural networks, but the first assignments are very general and will give a lot of insight of what and how neural networks are doing.