Origins of the name of convolutional neural networks

Question

Convolutional neural networks (CNNs) contain convolutional layers. In modern deep learning libraries such as Tensorflow and PyTorch, convolutional layers are implemented by using the cross-correlation operator instead of the convolution operator. The difference is that in convolution, the kernel is flipped before applying it to the input.

For example, in the book "Deep Learning", it is explained as follows.

Many machine learning libraries implement cross-correlation but call it convolution. --- In the context of machine learning, the learning algorithm will learn the appropriate values of the kernel in the appropriate place, so an algorithm based on convolution with kernel flipping will learn a kernel that is flipped relative to the kernel learned by an algorithm without the flipping. It is also rare for convolution to be used alone in machine learning; instead convolution is used simultaneously with other functions, and the combination of these functions does not commute regardless of whether the convolution operation flips its kernel or not.

This makes perfect sense and convincingly argues why implementing the flipping of the kernel would be unnecessary.

But how come CNNs are not commonly called "cross-correlational neural networks" instead of "convolutional neural networks"? To the best of my knowledge, the first concrete implementations of CNNs predate any of the above-mentioned libraries. Did these early implementations of CNNs indeed use the convolution operator, leading to the name? Or is there another reason?

Answer 1

It's inherited from math, but computer scientists optimized the algorithm and stuck with the original term.

3Blue1Brown also comes to the same conclusion (at 13:23) that

Another thing worth highlighting is that in the computer science context, this notion of flipping around that kernel before you let it march across the original often feels really weird and just uncalled for. But again, note that that's what's inherited from the pure math context, where like we saw with the probabilities, it's an incredibly natural thing to do.

One of the first mentions of 'convolutions' in neural networks is in the seminal paper by Yann Le Cun et al (1989). However, they also do not specify whether they perform true convolution. They simply state

This operation is equivalent to a convolution with a small size kernel, followed by a squashing function.

Answer 2

Maths and calculus. I remember studying convolution 40 years ago in the Digital Signal Processing subject at the University.

"It was the 1760s when the Swiss mathematical genius Leonhard Euler (1707–1783) suffered complete blindness, but this illness did not prevent him from contributing to mathematics. In fact, in those years, he wrote a memorable book about integral calculus and the solution of differential equations (DEs) by certain definite integrals [6]. ....

A first and simple case of this solution is K(u) = u and Q(x) = x and a second case is K(u) = -u and Q(x) = x. Euler studied the first case in [7], which is the first source in which the correlation operation is used exhaustively in statistical applications and related to convolution. "

A History of the Convolution Operation