Is it possible to give a rule of thumb estimate about the size of neural networks that are trainable on common consumer-grade GPUs?
For example, the Emergence of Locomotion (Reinforcement) paper trains a network using tanh activation of the neurons. They have a 3 layer NN with 300,200,100 units for the Planar Walker. But they don’t report the hardware and time.
But could a rule of thumb be developed?
Also, just based on current empirical results. So, for example, $X$ units using sigmoid activation can run $Y$ learning iterations per hour on a 1060.
Or using activation function $a$ instead of $b$ causes a $n$ times decrease in performance.
If a student/researcher/curious mind is going to buy a GPU for playing around with these networks, how do you decide what you get? A 1060 is apparently the entry-level budget option, but how can you evaluate if it is not smarter to just get a crappy netbook instead of building a high-power desktop and spend the saved $ on on-demand cloud infrastructure.
Motivation for the question: I just purchased a 1060 and (clever, to ask the question afterwards huh) wonder if I should have just kept the $ and made a Google Cloud account. And if I can run my master thesis simulation on the GPU.