# why learn an observation model when training latent space model in model based rl

I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley.

At 1:52 of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model $$p(o_t | s_t)$$ instead of a state model $$p(s_t | o_t)$$? For me, it seems more intuitive to get a state from a state model $$p(s_t|o_t)$$ and use a state transition model to plan. Is there a specific reason to learn an observation model instead of a state model?

Also at 4:38 of the lecture, following objective function is presented.

$$max_\phi \frac{1}{N} \Sigma_{i=1}^{N} \Sigma_{t=1}^{T} E_{(s_t, s_{t+1}) \sim p(s_t, s_{t+1}|o_{1:T}, a_{1:T})}[logp_\phi(s_{t+1,i} | s_{t,i}, a_{t, i}) + logp_\phi(o_{t, i} | s_{t, i})]$$

In the objective function, $$logp_\phi(o_t | s_t)$$ is included. I understand that maximizing this term could be desirable since it can be interpreted as minimizing a reconstruction loss. However, I cannot clearly see why $$p_\phi(o_t|s_t)$$ would be preferable over $$p_\phi(s_t|o_t)$$ as an objective function term.

I tried to interpret $$p_\phi(o_t|s_t)$$ term in the objective function as a maximum log likelihood and thus justify having it in the loss instead of $$p_\phi(s_t|o_t)$$. However, I do not think that it can be interpreted as a log likelihood since $$E_{s_t \sim p_\phi(s_t|o_t)}[logp_\phi(o_{t, i} | s_{t, i})] = \Sigma_{s_{t,i}} logp_\phi(o_{t, i} | s_{t, i}) * p_\phi(s_{t,i}|o_{t,i}) \neq logp_\phi(o_{t,i})$$

In summary, my questions are following.

1. Why learn an observation model $$p_\phi(o_t|s_t)$$ instead of a state model $$p_\phi(s_t|o_t)$$
2. Why have the reconstruction term in the loss instead of $$logp_\phi(s_t|o_t)$$?

Thank you

The problem is the definition of what's $$o_t$$ and $$s_t$$:

• $$o_t$$ is the (partially) observable part
• $$s_t$$ is ideally a perfect model of the world

now, by definition, you don't have access directly to $$s_t$$, otherwise half of the problem would be solved on the fly

2. because you want to learn a state representation that carries information about your future, which is $$o_t$$ (think about the RNN encoder, I give you the first N frames of a video, you give me the N+1 frame, so the last "latent vector" $$s_t$$ should carry information useful to predict what's about to happen $$o_t$$)