The ability to recognize an object with particular identifying features from single or multiple camera shoots with the temporal dimension digitized as frames has been shown. The proof is that the movie industry does face replacement to reduce liability costs for stars when stunts are needed. It is now done in a substantial percentage of action movie releases.
This brings up the question of how valuable recognizing a stop sign is compared to the value of recognizing an action. For instance, in the world of autonomous vehicles, should there even be stop signs. Stop signs are designed for lack of intelligence or lack of attention, which is why any police officer will tell you that almost no one comes to a full stop per law. What human brains intuitively looks for is the potential of collision.
Once what we linguistically perceive as verbs can be handled in deep learning scenarios as proficiently as nouns can be handled, the projection of risk becomes possible.
This may be very much the philosophy behind the proprietary technology that allows directors to say, "Replace the stunt person's face with the movie's protagonist's face," and have a body of experts execute it using software tools and LINUX clusters. The star's face is projected into the model of the action realized in the digital record of the stunt person.
Projected action is exactly what our brain does when we avoid collisions, and not just with driving. We do it socially, financially, when we design mechanical mechanisms, and in hundreds of other fields of human endeavor.
If we consider the topology of GANs as a loop in balance, which is what it is, we can then see the similarity of GANs to the chemical equilibria between suspensions and solutions. This gives us a hint into the type of topologies that can project action and therefore detect risk from audiovisual data streams.
Once action recognition is mastered, it is a smaller step to use the trained model to project the next set of frames and then detect collision or other risks. Such would most likely make possible a more reliable and safe automation of a number of AI products and services, breaking through a threshold in ML, and increased safety margins throughout the ever increasing world population density.
... which brings us back to ...
What topologies support recognition of action sequences?
The topology may have convolution, perhaps in conjunction with RNN techniques, encoders, equilibria such as the generative and discriminative models in GANs, and other design elements and concepts. Perhaps a new element type or concept will need to be invented. Will we have to first recognize actions in a frame sequence and then project the consequences of various options in frames that are not yet shot?
Where would the building blocks go and how would they be connected, initially dismissing concerns about computing power, network realization, and throughput for now?
Work may have been done along this area and realized in software, but I have not seen that degree of maturity yet in the literature, so most of it, if there is any, must be proprietary at this time. It is useful to open the question to the AI community and level the playing field.
