I will first explain briefly to you the difference between supervised learning and reinforcement learning to make sure that you don't have any misunderstandings. In supervised learning you are provided with some data $\{(\textbf{x}_i, y_i)\}_{i=1}^n$ where $\textbf{x}_i$ are the features for data point $i$ and $y_i$ is its true label. Now, the aim of supervised learning is to learn a function $f$ that can accurately predict the label of a data point given its features. In deep learning this function is a neural network, $f_\theta(\cdot)$. To optimise the parameters we obtain the models prediction for the label of $y$, denoted typically by $\hat{y} = f_\theta(x)$ and we look to optimise the parameters of the function $\theta$ by minimising the loss $\mathcal{L}(\hat{y}, y)$ (note that here the loss is a function $\mathcal{L}: \mathcal{Y} \times \mathcal{Y} \rightarrow \mathbb{R}$.
In reinforcement learning things are quite different. We are not provided with any data. Instead we have a Markov Decision Process (MDP) and an environment that we can interact with. The state space is defined by the MDP and we can get samples, mainly tuples of the form $(s, a, r, s')$, that we can use to teach our agent how to find an optimal policy - typically an optimal policy is one that takes an action given the current state that will maximise the sum of the future rewards of the episode. So in reinforcement you can probably see that things are quite different to supervised learning, mainly in the way that we 'obtain' our data.
Of course these explanations are gross oversimplifications of the learning process in both paradigms and should only be used as a brief example to try to emphasise the different between the two learning paradigms.
Now, there are two ways you can parameterise your Q-function using a neural network:
- The network takes as input the current state and outputs Q-values for each potential action; i.e. the output is a vector in $\mathbb{R}^{|\mathcal{A}|}$;
- The network takes as input the state and action and outputs a real number which is the Q-value for the state, action tuple you pass as input to the network.
The second way is usually reserved for instances where you have a huge action space but only ever would consider a few feasible actions at each state - this saves computational complexity.
Now, to answer your question:
In case 1) you are assumed to have access to a transition tuple $(s, a, r, s')$. The temporal difference, which we will use as the target, is $\hat{y} = r + \max_a Q(s', a)$. Now, as our output of $Q(s', \cdot)$ is an $\mathbb{R}^{|\mathcal{A}|}$ what we do is make a forward pass of the network for the current $(s, a)$ tuple, i.e. we get $x = Q(s, a)$, and then change the element that corresponds to the action which satisfies $\arg\max_a Q(s', a)$ of $x$ to $\hat{y}$, so we have now got our augmented input $\tilde{x}$ which serves as our target.
To make that step a bit clearer, suppose we have a 2-dimensional action space and the $\arg\max_a Q(s', a)$ is the action in the first dimension, then we would change the first dimension of $x$ to be $\hat{y}$.
We then train the network using the Mean Squared Error Loss between $x$ and $\tilde{x}$ - note that no gradient information is retained when we do the forward pass to get $\tilde{x}$; in fact we usually don't use a current version of the $Q$ network, we use an 'old' version of the network called the target network (I imagine there's probably a question about this network on the site already so I won't explain it in detail).
In case 2) the idea is much more simple as the outputs of the network are scalars so you can just train your network using the MSE between the scalar values of $Q(s, a)$ and $\hat{y}$ as defined above, the caveat here is that to calculate $\arg\max_a Q(s', a)$ you have to make $|\mathcal{A}|$ forward passes of the network for all possible $(s', a)$, for fixed $s'$, tuples which is why method 1) is usually preferred.
Now, whilst we may use techniques from supervised learning to optimise the Q-function, you can hopefully see the differences between supervised and reinforcement learning.
In short, the temporal difference is the target that you train your network towards being able to predict, and the state is (typically) the input into your neural network. For a more in depth discussion of RL being framed as supervised learning please see this answer.
