# How is the DQN able to generalise the learning to unseen states with such a loss function?

I am trying to understand how deep Q learning (DQN) works. To my current understanding, each $$Q(s, a)$$ functions is estimated to be a function of a feature vector of its state $$\phi$$(s) and the weight of the network $$\theta$$.

The loss function to minimise is $$||\delta_{t+1}||^2$$ where $$\delta_{t+1}$$ is shown below. The loss function is from the website talking about function approximation. Even though it is not explicitly deep Q learning, the loss function to minimise is similar.

$$\delta_{\mathrm{t}+1}=\mathrm{R}_{\mathrm{t}+1}+\max _{\mathrm{a}\in\mathrm{A}} \boldsymbol{\theta}^{\top} \Phi\left(\mathrm{s}_{t+1}, \mathrm{a}\right)-\boldsymbol{\theta}^{\top} \Phi\left(\mathrm{s}_{\mathrm{t}}, \mathrm{a}\right)$$

Intuitively, I am not able to understand why the loss function is defined as such. Once the network converges to a $$\theta$$ using gradient descent, does that mean that the $$Q_{max}(s,a)$$ is found?

In essence, I am not able to grasp intuitively how the neural network is able to generalise the learning to unseen states.

The algorithm I am looking at to help me understand the deep Q networks is below.

• I've tried to rewrite the title so that it contains what I think is the actual question. Make sure that the title contains what was your actual question.
– nbro
Apr 14 at 14:41