What you want to look for is called anaphora resolution. You basically keep a record of the past conversation and try and find an antecedent for any occurrences of it, he/she, her/his, etc. You probably want to have a pre-processing step where you substitute the antecedent before passing the input sentence on to the agent.
Have a look at Named Entity Recognition (NER); these algorithms are mainly concerned with recognising that there is an entity, but often also include normalising the name to a canonical form for information retrieval -- this is what you would need.
In a previous job I actually implemented this, using a fuzzy match with variable word order. You would still ...
Disclaimer: Without the full code, we can only speculate. I encourage you to post the full code on Google Colab or something like this.
In the meanwhile, here is my point of view:
Looks like your model has found some "master action" that always leads to zero loss, no matter what the state is. So it's not necessarily bad, it's just ...