I've been reading this paper that formulates invariant task-parametrized HSMMs. The task parameters are represented in $F$ coordinate systems defined by $\{A_j,b_j\}_{j=1}^F$, where $A_j$ denotes the rotation of the frame as an orientation matrix and $b_j$ represents the origin of the frame. Each datapoint $\xi_t$ is observed from the viewpoint of $F$ different experts/frames, with $\xi_t^{(j)} = A_j^{-1}(\xi_t - b_j)$ denoting the datapoint w.r.t. frame $j$. I quote from the abstract:
"Generalizing manipulation skills to new situations requires extracting invariant patterns from demonstrations. For example, the robot needs to understand the demonstrations at a higher level while being invariant to the appearance of the objects, geometric aspects of objects such as its position, size, orientation and viewpoint of the observer in the demonstrations."
"The algorithm takes as input the demonstrations with respect to different coordinate systems describing virtual landmarks or objects of interest with a task-parameterized formulation, and adapt the segments according to the environmental changes in a systematic manner."
Though it makes some intuitive sense, I'm not fully convinced why working with multiple coordinate systems would help us capture invariant patterns in demonstrations, and leave aside the scene-specific details. That is the goal, right? On a very high level, I see that having access to more "viewpoints" may help the robot understand the environment better, and neglect viewpoint-specific biases to focus on invariant patterns across different frames. However, this is very handwavy - and I'd love to know specific details about why using multiple viewpoints is a good idea in this case.
Thanks!
