1
$\begingroup$

I have seen this encoding of an image as a graph:

  • The set of the nodes $V$ is the set of pixels. If the image is of size $10\times10$, then we have $10\cdot10=100$ pixels.
  • Each node has a length 3 feature vector $x_i = (r,g,b)$ which states the Red, Green and Blue intensity for each pixel.
  • All nodes are connected. Furthermore, an adjacency matrix $A$ contains at the element $(i,j)$ the distance between pixel $i$ and $j$ encoded as a number between zero and one. For instance, $A_{2,2}=0$ and $A_{firstpixel, lastpixel}=1$

Is this better for any reason, in any context? Maybe using this in GCN makes sense for some applications?

I honestly don't see the point. GCNs are used for either link prediction or node classification. In this case, the nodes have no classification (they're just pixels) and the link is just the distance between the pixels, and there's of course no interest in predicting links here.

Is my intuition wrong?

Improve this question
$\endgroup$
3
  • $\begingroup$ How would you want to use an image in a graph neural network? I don't understand if you're asking if it makes sense to pass images to graph neural networks or why do we encode images as graphs to pass to GNNs. The first question makes more sense to me than the latter. $\endgroup$
    – nbro
    Commented Feb 3, 2023 at 11:19
  • $\begingroup$ @nbro This is actually done either by treating the image as a regular grid graph, or by subdividing it into super-pixels and treating this as a graph. However, I've seen this mostly in Graph Benchmarking and not really if predictive performance is required. $\endgroup$
    – Chillston
    Commented Feb 3, 2023 at 13:35
  • $\begingroup$ @Chillston Yes, right. Sorry, the formulation of my first question in the comment above is not correct. My question was more: how do you think you can pass an image to a GNN otherwise? If GNNs process graphs, then images need to be converted to graphs. Anyway, thanks for the comment. I am also aware of this, although it's been a while since I had to do something with GNNs. $\endgroup$
    – nbro
    Commented Feb 3, 2023 at 13:51

1 Answer 1

Reset to default
0
$\begingroup$

I've seen what you are describing mostly in GNN benchmarking and in that context it makes sense in order to gauge the model performance on regular graphs (which can be challenging for GCNs that are based on the Weisfeiler-Lehman-Kernel; cf. Douglas 2011 and Xu et al. 2019).

However, to get performance for image based tasks I don't think that there is a scenario where GNNs would be more suitable than CNNs. But I would extend your argument: GNNs can be used for graph classification just as well as for node and edge classification (this gets clear when you search for GNNs that perform inductive tasks like molecule classification).

I would argue that CNNs outperform GNNs because CNNs are specified to regular grids, whereas GNNs are the more general model that can be applied to arbitrary structures. This is due to the design of the kernel in these models. Whereas GNNs apply a permutation invariant kernel, CNNs are sensitive to permutation. Thus the CNN kernel is sensitive to things like orientation and reflection. The GNN kernel is not, which intuitively is a clear disadvantage in the image case.

A point where GNNs can be useful in the context of computer vision is when involving the scene graphs, because these are irregular structures that cannot be modeled by a CNN.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .