I'm new to the field of ML so please bear with me while I try to explain what I'm looking for. In most machine learning pipelines that deal with images there is a requirement to "normalize" the data in some way so that images of different dimensions can be used as inputs for the function that is being optimized. As in, if the function takes its input as an $n\times n$ grid of pixels (assuming we're dealing with 1-channel images) then any image that is not of the right shape must be re-shaped so that it can be used as input. We can assume $n = 2$ without losing any generality because any larger image can be reduced to the $2\times 2$ case for what I'm about to describe.

So if we assume we have a $2\times 2$ image then there is an obvious way to map such an image to a function defined on $[0,1]\times [0,1]$ ($f:[0,1]\times[0,1]\rightarrow\mathbb{R}$) by using convex combinations of the points in the image. If the points of the image are labeled as $x_{00},x_{01},x_{10},x_{11}$ where $x_{00}$ is the top left corner and $x_{11}$ is the bottom right corner then given a point $(a,b)\in[0,1]\times[0,1]$ the value of $f$ at $(a, b)$ can be defined as $$f(a, b) = (1-b)((1-a)x_{10}+ax_{11})+b((1-a)x_{00}+ax_{01})$$

Assuming I got all the signs right it's obvious that this idea can be extended to any grid of pixels by mapping the horizontal and vertical dimensions to $[0,1]$ and then interpolating between the grid points as in the $2\times 2$ case. So this mapping from grids of pixels to functions provides a uniform representation for all images as functions defined on $[0,1]\times[0,1]$.

Now my question is the following: Is there any work that tries to use this kind of representation of images and if there isn't does anyone know what exactly are the obstructions to doing so? It's possible I'm missing something that makes this approach non-viable but I wasn't able to find anything that explained one way or the other why the usual tensor representation is preferable to a functional one as above that reduces all images to functions on $[0,1]\times[0,1]$.

  • $\begingroup$ You seem to be trying to describe bilinear interpolation, I think? $\endgroup$
    – user253751
    Sep 20 at 12:21
  • $\begingroup$ Yes, this is bilinear interpolation followed by rescaling to make everything fit into the unit square. It's obvious to me that this representation subsumes the usual tensor one because every tensor representation can be reduced to a functional one by adding the dimensions as another set of inputs to the learning algorithm, e.g. $(w, h, f)$ where $w$ is the width and $h$ is the height of the image. The question is why aren't there more learning algorithms structured around this representation? $\endgroup$ Sep 20 at 17:06
  • $\begingroup$ To explain a bit more. Images have a topology that the usual tensor representation does not account for because it treats each pixel independently from its neighbors but it's obvious that there is a continuity in natural images because pixels that are close together often have similar values but the tensor representation does not account for this natural topology. In the representation I'm proposing this natural continuity is made obvious because the function that represents the image is continuous so it has more structure that the tensor representation doesn't. $\endgroup$ Sep 20 at 17:17
  • $\begingroup$ Why do you think the tensor representation doesn't account for this natural topology? Pixels that are close together in the image are also close together in the tensor, and convolutional layers take advantage of that (as well as the fact that we want to look for the same thing in any part of the image) $\endgroup$
    – user253751
    Sep 20 at 21:16
  • $\begingroup$ Can you describe the topology of the tensor representation? The topology of both the unit square and $\mathbb{R}$ are the standard ones taught in most undergrad math classes with $\epsilon$ neighborhoods. What is the equivalent for the tensor/matrix representation? $\endgroup$ Sep 20 at 23:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.