why learn an observation model when training latent space model in model based rl

Question

I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley.

At 1:52 of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model $p(o_t | s_t)$ instead of a state model $p(s_t | o_t)$? For me, it seems more intuitive to get a state from a state model $p(s_t|o_t)$ and use a state transition model to plan. Is there a specific reason to learn an observation model instead of a state model?

Also at 4:38 of the lecture, following objective function is presented.

$max_\phi \frac{1}{N} \Sigma_{i=1}^{N} \Sigma_{t=1}^{T} E_{(s_t, s_{t+1}) \sim p(s_t, s_{t+1}|o_{1:T}, a_{1:T})}[logp_\phi(s_{t+1,i} | s_{t,i}, a_{t, i}) + logp_\phi(o_{t, i} | s_{t, i})]$

In the objective function, $logp_\phi(o_t | s_t)$ is included. I understand that maximizing this term could be desirable since it can be interpreted as minimizing a reconstruction loss. However, I cannot clearly see why $p_\phi(o_t|s_t)$ would be preferable over $p_\phi(s_t|o_t)$ as an objective function term.

I tried to interpret $p_\phi(o_t|s_t)$ term in the objective function as a maximum log likelihood and thus justify having it in the loss instead of $p_\phi(s_t|o_t)$. However, I do not think that it can be interpreted as a log likelihood since $E_{s_t \sim p_\phi(s_t|o_t)}[logp_\phi(o_{t, i} | s_{t, i})] = \Sigma_{s_{t,i}} logp_\phi(o_{t, i} | s_{t, i}) * p_\phi(s_{t,i}|o_{t,i}) \neq logp_\phi(o_{t,i})$

In summary, my questions are following.

1. Why learn an observation model $p_\phi(o_t|s_t)$ instead of a state model $p_\phi(s_t|o_t)$
2. Why have the reconstruction term in the loss instead of $logp_\phi(s_t|o_t)$?

Thank you

Answer 1

The problem is the definition of what's $o_t$ and $s_t$:

• $o_t$ is the (partially) observable part
• $s_t$ is ideally a perfect model of the world

now, by definition, you don't have access directly to $s_t$, otherwise half of the problem would be solved on the fly

So, answering your questions:

1. Because there is no way to learn it, instead as proposed in the video, you use a NN to approximate a posterior, which takes as input all the previous observations and actions, and gives you a useful state representation (think about an RNN as encoder, the final latent vector is the state representation)
2. because you want to learn a state representation that carries information about your future, which is $o_t$ (think about the RNN encoder, I give you the first N frames of a video, you give me the N+1 frame, so the last "latent vector" $s_t$ should carry information useful to predict what's about to happen $o_t$)