Convolutional neural networks (CNNs) contain convolutional layers. In modern deep learning libraries such as Tensorflow and PyTorch among others, convolutional layers are implemented by using the cross-correlation operator instead of the convolution operator. The difference is that in convolution, the kernel is flipped before applying it on the input.

For example in the book "Deep Learning", it is explained as follows.

Many machine learning libraries implement cross-correlation but call it convolution. --- In the context of machine learning, the learning algorithm will learn the appropriate values of the kernel in the appropriate place, so an algorithm based on convolution with kernel flipping will learn a kernel that is flipped relative to the kernel learned by an algorithm without the flipping. It is also rare for convolution to be used alone in machine learning; instead convolution is used simultaneously with other functions, and the combination of these functions does not commute regardless of whether the convolution operation flips its kernel or not.

This makes perfect sense, and convincingly argues why implementing the flipping of the kernel would be unnecessary.

But how come CNNs are not commonly called "cross-correlational neural networks" instead of "convolutional neural networks"? To the best of my knowledge, the first concrete implementations of CNNs predate any of the above mentioned libraries. Did these early implementations of CNNs indeed use the convolution operator, leading to the name? Or is there another reason?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.