I am trying to classify actions in untrimmed videos. These videos contain a very imbalanced set of actions (where the background class is the majority). I have previously tried frame-wise action classifications with Temporal Convolutional Neural Networks, but these didn't work great and tended not to localize the actions well. Therefore, I thought this method is inflexible and turned to temporal action localisation methods to detect the temporal segments of the various actions from within the background class. See BMN: Boundary-Matching Network for Temporal Action Proposal Generation.
Is there any method in the literature that combines temporal action proposals with multiple actions' classifiers?
What seems to be the case (I might be wrong) is that these proposals are applied to videos where only one action is present multiple times in the untrimmed videos, so the combination is simply a form of weak supervision where only the video label is provided.