I have once tried Viola Jones Algorithm to do that, it does not capture the subtle differences in the direction of facial segments which are important to detect emotion. Features like HOG (Available in openCV and many famous image processing libraries) can extract better information from the face to classify emotion.
Also there are many other approaches including ANNs and pure rule based approaches But almost everywhere a good alignment approach for faces become the most important aspect of the exercise. So I will suggest exploring some facial alignment approaches and then Features like HOG instead of Viola Jones/ HARR.
For the mathematics part, it is upto you to dive deep into mathematics or just exploring different approaches by codes. A good understanding of Linear Algebra and a little Geometry will help a lot.
Also if you are new to Machine Learning, understanding the basic algorithms might be relevant to you.