2
$\begingroup$

My question is pretty much the one asked above. To clarify a bit further: I have only found datasets that do object localization and that also have relations between the objects annotated (like: "Here is the horse, here is the rider. The rider rides the horse". What I am looking for is a dataset where you can do classification based on spatial relations (Like: "I am a positive example, because the rider is above the horse. I am a negative example, because the rider is below the horse (or besides it, etc)").

Any help is appreciated

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

There is no specific image classification dataset that focuses on spatial relations. However, there are some datasets that include images with spatial relations annotations, such as the Visual Relationship Detection (VRD) dataset which is a collection of images with annotations of spatial relations between objects.

Here is a paper Identifying Spatial Relations in Images using Convolutional Neural Networks that uses pre-trained CNN to learn spatial relations between objects in an image, it makes use of SUN09 and a simplified synthetic dataset to trained the network. the network pays attention to a certain parts of images while classifying them for spatial relations.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ Thank you so much for your answer! I already had a look into the VRD dataset and I think it is suitable for learning relations. But my problem is a bit different in that I do not need to learn the spatial relations (my approach looks at important image parts and finds spatial relations just by the coordinates. Do you think it would be possible to get a subset from such datasets where the ground truth of an image depends on the relations? Thank you also for the linked paper as it gives me further ideas for my research. $\endgroup$
    – Johannes R.
    Oct 17, 2022 at 14:11
  • 3
    $\begingroup$ Yes, I think it would be possible to get a subset from such datasets where the ground truth of an image depends on the relations. However, it is not clear how useful this would be, since the goal of the datasets is to learn generalizable models from data, and not to create models that only work on a specific subset of data. $\endgroup$
    – Faizy
    Oct 17, 2022 at 14:55
  • 2
    $\begingroup$ Second, the ground truth of an image may not be entirely determined by the relations between objects in the scene, but may also be influenced by other factors such as lighting or the position of the camera. $\endgroup$
    – Faizy
    Oct 17, 2022 at 15:08
  • 2
    $\begingroup$ Thanks for pointing that out. I think I need to explain my specific case a bit further: I am researching on Explainable AI that goes beyond visual explanations like LIME, GradCAM etc. and build explanations based on first order logic (like: "the image belongs to this class since the important objects A, B, C are in a particular spatial relation to each other"). I am aware that this topic might be very new and so there is not a lot of datasets that are usable for this particular purpose. At least no sets that go beyond toy datasets. $\endgroup$
    – Johannes R.
    Oct 17, 2022 at 18:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .