3
$\begingroup$

I was thinking about different neural network topologies for some applications. However, I am not sure how this would affect the efficiency of hardware acceleration using GPU/TPU/some other chip.

If, instead of layers that would be fully connected, I have layers with neurons connected in some other way (some pairs of neurons connected, others not), how is this going to affect the hardware acceleration? An example of this is the convolutional networks. However, there is still a clear pattern, which perhaps is exploited by the acceleration, which would mean that if there is no such pattern, the acceleration would not work as well?

Should this be a concern? If so, is there some rule of thumb for how the connectivity pattern is going to affect the efficiency of hardware acceleration?

$\endgroup$
  • $\begingroup$ Neural network optimization is equal to meta-learning. The idea is not only reduce the error value of the given neural network structure, but to improve the network itself, which includes to test out new topologies. The algorithm of choice is a genetic algorithm, which is sometimes called neuro evolution. neat $\endgroup$ – Manuel Rodriguez Sep 23 at 16:00
  • 1
    $\begingroup$ Thank you for your response, but it does not answer the question. When I change the way layers are connected from fully connected to only a subset of pairs of neurons being connected, how is it going to change the benefit that I can get from GPU/TPU. $\endgroup$ – user2316602 Sep 23 at 18:33
  • 1
    $\begingroup$ @user2316602: The answer is going to depend on details - e.g. how sparse the connections are, whether they can be arranged meaningfully into layers etc. Also whether you have training data available in large batches. Do you have any specific use case or scenario that would narrow the scope down a little? Or are you looking for a broad but shallow answer? $\endgroup$ – Neil Slater Sep 23 at 19:00
  • $\begingroup$ I am interested in a broad but shallow answer. Or even better, to some place where I could read more. $\endgroup$ – user2316602 Oct 3 at 18:02
-1
$\begingroup$

Convolutional networks can actually be trained in parallel very well. Even Recurrent neural networks RNNs that can't be parallelized that well, have a CUDA implementation in tensorflow and pytorch that is optimized to run on a GPU and which performs from experience as well a CNN.

$\endgroup$
  • 1
    $\begingroup$ Thank you for your response, but it does not answer the question. When I change the way layers are connected from fully connected to only a subset of pairs of neurons being connected, how is it going to change the benefit that I can get from GPU/TPU. $\endgroup$ – user2316602 Sep 23 at 18:32
  • $\begingroup$ I'm only aware of the effect on the most common architectures that I mentioned. For more details about those three architectures, you can have a look at this medium.com/@culurciello/… $\endgroup$ – cookiemonster Nov 4 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.