1
$\begingroup$

So I built a CNN without any scientific libraries like TensorFlow or Keras (only NumPy). It is taking a huge amount of time to train. What are some of the tricks and tips followed by people to speed up training of a CNN? (I am not talking about division of jobs into different processors but subtle redundant codes i.e. giving pre-calculated results which is not visible to common programmers).

$\endgroup$
  • $\begingroup$ This is rather hard to answer without knowing more details... Have you tried profiling it to see where the bottlenecks are? Is it floating-point arithmetic? Data I/O? $\endgroup$ – Oliver Mason Jun 19 '18 at 11:50
  • $\begingroup$ @OliverMason I agree details are not there but there must be some standard checks performed by programmers...this is more of a general open ended question for future references rather than code specific $\endgroup$ – DuttaA Jun 19 '18 at 11:51
  • $\begingroup$ @OliverMason what do you mean by profiling? (I am not conversant with programming terminology) $\endgroup$ – DuttaA Jun 19 '18 at 11:52
  • $\begingroup$ Measure where the time is being spent within the program. Then you can focus on those parts, rather than spend lots of time speeding up something that is fast anyway. $\endgroup$ – Oliver Mason Jun 19 '18 at 12:59
  • $\begingroup$ Why are you even caring about the speed of a hand-rolled, non-accelerated CNN implementation? You only build those to understand what's happening. The goal in insight, not performance. $\endgroup$ – MSalters Jun 20 '18 at 9:29
1
$\begingroup$

[Ref-some standard checks performed by programmers]

Speeding up Convolutional Neural Networks with Low Rank Expansions

From the abstract:

The focus of this paper is speeding up the evaluation of convolutional neural networks. While delivering impressive results across a range of computer vision and machine learning tasks, these networks are computationally demanding, limiting their deployability.

Convolutional layers generally consume the bulk of the processing time, and so in this work we present two simple schemes for drastically speeding up these layers. This is achieved by exploiting cross-channel or filter redundancy to construct a low rank basis of filters that are rank-1 in the spatial domain.

Our methods are architecture agnostic, and can be easily applied to existing CPU and GPU convolutional frameworks for tuneable speedup performance. We demonstrate this with a real world network designed for scene text character recognition, showing a possible 2.5x speedup with no loss in accuracy, and 4.5x speedup with less than 1% drop in accuracy, still achieving state-of-the-art on standard benchmarks.

$\endgroup$
  • $\begingroup$ any feedback @DuttaA? $\endgroup$ – Guilherme IA Jun 21 '18 at 19:01
  • 1
    $\begingroup$ @GuilhermeIA currently my plate is full...for now i think i'll bypass the problem using tensorflow, but within a few days i'll have to write my own code...i promise i'll read the paper then and probably give a gist in the answer or accept the answer if it provided some answer and i'll tag you :) $\endgroup$ – DuttaA Jun 22 '18 at 2:24
1
$\begingroup$

Recommendations:

  • Try deleting some variables that will no longer be used during run-time
  • Use more efficient data structure
  • Get your hands on some optimized library for your hardware, e.g. if you are using Intel processors use the Intel distribution of python
  • Pay careful attention to your data types and try to trim them as much as possible
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.