The basis of my question is that a CNN that does great on MNIST is far smaller than a CNN that does great on ImageNet. Clearly, as the number of potential target classes increases, along with image complexity (background, illumination, etc.), the network needs to become deeper and wider to be able to sufficiently capture all of the variation in the dataset. However, the downside of larger networks is that they become far slower for both inference and backprop.
Assume you wanted to build a network that runs on a security camera in front of your house. You are really interested in telling when it sees a person, or a car in your driveway, or a delivery truck, etc. Let's say you have a total of 20 classes that you care about (maybe you want to know minivan, pickup, and so on).
You gather a dataset that has plenty of nice, clean data. It has footage from lots of times of the day, with lots of intra-class variation and great balance between all of the classes. Finally, assume that you want this network to run at the maximum possible framerate (I know that security cameras don't need to do this, but maybe you're running on a small processor or some other reason that you want to be executing at really high speed).
Is there any advantage, computationally, to splitting your network into smaller networks that specialize? One possibility is having a morning, an afternoon/evening, and a night network and you run the one corresponding to the time of day. Each one can detect all 20 classes (although you could split even farther and make it so that there is a vehicle one, and a person one, and so on). Your other option is sharing base layers (similar to using VGGNet layers for transfer learning). Then, you have the output of those base layers fed into several small networks, each specialized like above. Finally, you could also have just one large network that runs in all conditions.
Question: Is there a way to know which of these would be faster other than building them?
In my head, it feels like sharing base layers and then diverging will run as slow as the "sub-network" with the most additional parameters. Similar logic for the separate networks, except you save a lot of computation by sharing base layers. Overall, though, it seems like one network is probably ideal. Is there any research/experimentation along these lines?
