Let's take a 32 x 32 x 3 NumPy array and convolve with 10 filters of size 2 x 2 x 3 with stride 2 to produce feature maps of volume 16 x 16 x 10. The total number of operations - 16 * 16 * 10 * 2 * 2 * 2 * 3 = 61440 operations. Now, let's take an input array of length 3072 (flattening the 32 * 32 * 3 array) and dot it with a weight matrix of size 500 x 3072. The total number of operations - 500 * 3072 * 2 = 3072000 operations. The convolution takes 4-5 times longer than np.dot(w, x) even though number of operations is less.

Here's my code for the convolution operation:

for i in range(16):
    for j in range(16):
        for k in range(10):
            v[i, j, k] = np.sum(x[2 * i:2 * i + 2, 2 * j:2 * j + 2] * kernels[k]) 

Is np.dot(w, x) optimized or something? Or are my calculations wrong? Sorry if this is a silly question...

  • $\begingroup$ Programming questions are generally off-topic here. Please, ask this type of question on Stack Overflow. Here we focus on theoretical, philosophical and social aspects of AI. Please, read also our on-topic page ai.stackexchange.com/help/on-topic to know more about it. $\endgroup$ – nbro Sep 6 at 20:43

From the NumPy Linear algebra documentation:

The NumPy linear algebra functions rely on BLAS and LAPACK to provide efficient low level implementations of standard linear algebra algorithms. Those libraries may be provided by NumPy itself using C versions of a subset of their reference implementations but, when possible, highly optimized libraries that take advantage of specialized processor functionality are preferred.

In your case, np.dot() is a matrix-vector multiplication which internally calls such a highly optimized BLAS routine. Those libraries are implemented in low-level languages like Fortan or C, they exist for many years now and are still unbeaten in terms of speed.
Your naive implementation of the convolution operation in python on the other hand is not optimized for execution speed and will be executed by the python interpreter (which is much slower than the compiled C functions).

You could try to replace your manual convolution by the SciPy's equivalent scipy.signal.convolve2d(), to make advantage of optimized libraries as well and get a speed up.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.