3
$\begingroup$

I want to know if it is possible to train a neural network (or some other kind of an AI) to bring a simple picture story in the correct order, if it is in random order, so that the story has the correct story flow.

For example, this simple picture story:

enter image description here

or this one

enter image description here

So, imagine the pictures of these stories are in a random order and the AI has to put them in the order that the correct story is told.

Most 8 year olds would be able to do that. So, can an AI learn it? How would an approach look like? Does anyone know if something like that has been achieved or even tried?

From my research so far, the approach would be first to translate the images into descriptive sentences and then try to order them in a meaningful way. But I will do further research, I found so far this paper: Sort Story: Sorting Jumbled Images and Captions into Stories (2016).

To clarify, this is not a "real problem" for me, I just asked from a philosophical standpoint and from interest. I will not attempt to solve it, because I think if it is possible it would be extremely difficult.

$\endgroup$
1
  • $\begingroup$ I just was interested if an AI are already that advanced or not. I found related topics, but for continuous images like movements in a image stream like a video. This is no external source it's just a random example for a picture story. I mean a picture storys for childs. For example if the six pictures in my example where in random order, if an ai can bring them in the correct "story flow". So it has to recognize whats the meaning behind each picture and connect (order them) in a meaningful way. Another exmaple: i.pinimg.com/474x/c2/41/11/c24111650800563c899c4efd4269ccdd.jpg $\endgroup$
    – Perry45
    Apr 11, 2022 at 11:14

1 Answer 1

1
$\begingroup$

This is a really hard problem for statistical AI such as a neural network. The difficulty level is due to lack of grounding and common sense in what a neural network can process.

A neural network could feasibly label all objects in the example scenes, and even do pose estimation, detect activities and guess emotional state for the "actors". It can even create a vector representing the content of the image and convert to/from a caption for the image.

However, so far any structure or embedding that neural networks have produced is not amenable to reasoning, or common sense. Such embeddings can be translated into other representations, but lack "grounding" in the sense of a deeper understanding based on a more general model of the world. It is this understanding - e.g. a parent will be stressed if they think a child is missing (panel 2 from second example), and may then act to find them (panel 4) - that is missing and it is not at all clear how such a world model could be added to neural network training.

There are some neural network models that come close in different ways:

  • Large language models. Descriptive text often has a narrative structure, and language models like GPT-3 can easily produce stories as sophisticated as the example panels. In theory such a model could be used to analyse the likelihood of different series of static descriptions extracted by an image captioning system, and identify the highest likelihood story based on trying all combinations. I do not know if this has been attempted.

  • Video activity prediction. In the simpler world of immediate actions and consequences (as opposed to understanding inner state, motivation and narrative), predicting what happens next using a neural network is already possible. These predictions would be short term - In the first panel of the first example for example, a neural network might predict that the fish will go into the bucket. That doesn't mean the neural network models what a fish or a bucket actually are in enough detail to reason about this further, nor that it could extrapolate to the excited child looking at the fish in the bucket on panel 2.

$\endgroup$
1
  • $\begingroup$ Thanks for this great answer! I think the point is common sense. I also though if you have enough training data it could be trained to guess the likelihood of emotion transitions based on various factors or something. But in fact you would actually train some kind of common sense this way I think. $\endgroup$
    – Perry45
    Apr 11, 2022 at 19:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .