is it possible to give a rule of thumb estimate about the size of neural networks that are trainable on common consumer grade GPUs? For example:
The Emergence of Locomotion (Reinforcement) paper trains a network using tanh activation of the neurons. They have a 3 layer NN with 300,200,100 units for the Planar Walker. But they don’t report the hardware and time ...
But could a rule of thumb be developed? Also just based on current empirical results, so for example:
$X$ Units using sigmoid activation can run $Y$ learning iterations per hour on a 1060.
Or using activation function $a$ instead of $b$ causes a $n$ times decrease in performance.
If a student/researcher/curious mind is going to buy a GPU for playing around with these networks, how do you decide what you get? A 1060 is apparently the entry level budget option, but how can you evaluate if it is not smarter to just get a crappy netbook instead of building a high power desktop and spend the saved $ on on-demand cloud infrastructure.
Motivation for the question: I just purchased a 1060 and (clever, to ask the question afterwards huh) wonder if I should have just kept the $ and made a Google Cloud account. And if I can run my master thesis simulation on the GPU.