Softmax policy $\pi_\theta(s,a)$ is defined as $\frac{\exp{(\phi(s,a)^T \theta})}{\Sigma \exp{(\phi(s,a) ^T \theta) }}$, where the summation is over the action space. Taking log, this becomes  \log \...

I guess you could train your model with 10 different folds and in each fold calculate the average accuracy. So you would have 10 values - one corresponding to each fold. And then take the mean of all ...

I don't think there is a fixed threshold that differentiates between Shallow and Deep Learning, but I would say that a 2 layer NN should not be considered deep. But now-a-days, almost all NN ...

Here is my understanding: In trajectory sampling as the book describes it, we use the current policy on the simulator to get (next-state, action) pairs. The advantage is that if some states occur more ...

I guess it would always be better if you can reuse existing environments to make it work for yourself. Since most of the environment codes is anyway opensourced, you can always edit it to your liking. ...

The advantage is basically a function of the actual return received and a baseline. The function of the baseline is to make sure that only the actions that are better than average receive a positive ...

It might be the case that if you perform a large number of random rollouts, the "best action" as chosen by the agent without the domain knowledge, is same as the agent with the domain knowledge. I ...

I think most of it is correct. Q function(also called state-action value, or just action value): How good it is to be in a state S and perform action A while following policy π. It uses reward to ...

I don't understand your question very clearly. Q-value of a particular state-action pair (s,a) under policy $\pi$ is the total reward you would expect to collect if you start from the state s, take ...

$Q(s,a)$ denotes the $Q-value$ for the state-action pair. It means the expected returns if we start from state $s$, take action $a$, and act according to whatever policy we are currently following. ...