I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley.

At 1:52 of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model $p(o_t | s_t)$ instead of a state model $p(s_t | o_t)$? For me, it seems more intuitive to get a state from a state model $p(s_t|o_t)$ and use a state transition model to plan. Is there a specific reason to learn an observation model instead of a state model?

Also at 4:38 of the lecture, following objective function is presented.

$max_\phi \frac{1}{N} \Sigma_{i=1}^{N} \Sigma_{t=1}^{T} E_{(s_t, s_{t+1}) \sim p(s_t, s_{t+1}|o_{1:T}, a_{1:T})}[logp_\phi(s_{t+1,i} | s_{t,i}, a_{t, i}) + logp_\phi(o_{t, i} | s_{t, i})]$

In the objective function, $logp_\phi(o_t | s_t)$ is included. I understand that maximizing this term could be desirable since it can be interpreted as minimizing a reconstruction loss. However, I cannot clearly see why $p_\phi(o_t|s_t)$ would be preferable over $p_\phi(s_t|o_t)$ as an objective function term.

I tried to interpret $p_\phi(o_t|s_t)$ term in the objective function as a maximum log likelihood and thus justify having it in the loss instead of $p_\phi(s_t|o_t)$. However, I do not think that it can be interpreted as a log likelihood since $E_{s_t \sim p_\phi(s_t|o_t)}[logp_\phi(o_{t, i} | s_{t, i})] = \Sigma_{s_{t,i}} logp_\phi(o_{t, i} | s_{t, i}) * p_\phi(s_{t,i}|o_{t,i}) \neq logp_\phi(o_{t,i})$

In summary, my questions are following.

  1. Why learn an observation model $p_\phi(o_t|s_t)$ instead of a state model $p_\phi(s_t|o_t)$
  2. Why have the reconstruction term in the loss instead of $logp_\phi(s_t|o_t)$?

Thank you


1 Answer 1


The problem is the definition of what's $o_t$ and $s_t$:

  • $o_t$ is the (partially) observable part
  • $s_t$ is ideally a perfect model of the world

now, by definition, you don't have access directly to $s_t$, otherwise half of the problem would be solved on the fly

So, answering your questions:

  1. Because there is no way to learn it, instead as proposed in the video, you use a NN to approximate a posterior, which takes as input all the previous observations and actions, and gives you a useful state representation (think about an RNN as encoder, the final latent vector is the state representation)
  2. because you want to learn a state representation that carries information about your future, which is $o_t$ (think about the RNN encoder, I give you the first N frames of a video, you give me the N+1 frame, so the last "latent vector" $s_t$ should carry information useful to predict what's about to happen $o_t$)
  • $\begingroup$ @platoDev no prob, good luck with the course $\endgroup$
    – Alberto
    Commented Sep 14, 2023 at 17:24
  • $\begingroup$ Thanks to your answer, it now seems obvious to me that observation is the only thing that is observable thus a reconstruction term should appear in the objective function. However, I have come to another question. What would be the possible interpretation of maximizing Σ𝑙𝑜𝑔𝑝𝜙(𝑜_𝑡|𝑠_𝑡)∗𝑝𝜙(𝑠_t|𝑜_t) where summation is over s_t? I still cannot get a grasp of it. It certainly is not likelihood. $\endgroup$
    – platoDev
    Commented Sep 15, 2023 at 4:09
  • $\begingroup$ @platoDev can you please post a new question with more informations and reference from where you see this notation? thanks $\endgroup$
    – Alberto
    Commented Sep 15, 2023 at 10:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .