I would use a single network:
The essence of the question is whether or not doing all the classification work at once is more efficient than running individual classifiers for each stage.
The recent "You Only Look Once" algorithm ("YOLO") is based on the fact that the convolutional networks can reuse a lot of the interim calculations if you combine them into one. Because of this, they are able to perform real-time object detection on images across thousands of classes.
You can express your hierarchical classifier with YOLO (man, jacket and jacket colour classes). Depending on your needs, you might want to model the jacket colour as a scalar output of an approximate R,G,B value for the colour rather than having named classes for the colours.
This everything-at-once implementation gives you runtime efficiencies for the inference step and much faster training since the classes share common abstractions in the earlier layers of the net.
Details, the YOLO version 2 paper, and a cool demonstration video featuring James Bond are available here: https://pjreddie.com/darknet/yolo/
The paper itself, "YOLO9000", is available on Arxiv: https://arxiv.org/abs/1612.08242
