9
My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one.
It has no real impact on computation time, other than a slight increase (due to extra memory used by two networks). You could cache results of the target network I suppose, but it probably would not ...
6
DQN and AlphaZero do not share much in terms of implementation.
However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, value, policy, then these are interchangeable between DQN and AlphaZero. When it comes to implementation, and what each part of the system is doing, then ...
5
Let $Q^*(s, a)$ denote the "true" $Q$-value for a state-action pair $(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs $Q(s, a)$ values.
The problem you describe is basically that you have situations where $Q^*(s, a_1) = Q^*(s, a_2) + \epsilon$ for some very small value $\epsilon$, where $a_1 \neq a_2$.
...
5
More precisely: is DQNN applicable only when we have high translational invariance in our input(s)?
No, DQN (that is the commonly-used abbreviation by the way, there also is a dqn tag which you may wish to use) is not restricted to images or other kinds of inputs with those properties, it can be used with pretty much any kinds of inputs.
The DQN algorithm ...
4
Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value V(s) function is estimated separately from the state-action value Q(s, a). A new quantity, the advantage of an action, can then be defined as A(s, a) = Q(s, a) - V(s).
The Q function, however, measures the the value
of choosing a ...
4
The simplest thing to do when you make you first implementation of the agent, is to automate decisions like this, in order to keep representations and decisions simple.
However, if you want to explore tactics surrounding declaration, then I think the following applies:
There should be an initial round of actions where the agent may get to decide whether or ...
4
In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do not usually do this, but it does happen sometimes.
I guess that, from a theoretical point of view, it was very easy of the authors to make their algorithm more ...
3
Does it have to do with the reward function?
This seems likely to me. You have chosen a reward that is unusual in that it cross-links episodes. It is not really a reinforcement problem to optimise behaviour with respect to results of previous episode behaviour in this way. This might be an option for an evolutionary fitness context, if you have competing ...
3
120 inputs can be handled by a complex enough network. Dealing with high complexity is one of NN's strengths.
Using a (120,) vector or a (3,40) matrix is the same, they're still 120 inputs. Your binary encoding should work. Another option is a single (40,) vector, with 0 being "still in deck", 1 being "in hand", 2 being "on table", 3 being "already played". ...
3
Although what @Jaden said may be true by itself, it does not really serve to answer my question as I have seen after conducting numerous experiments, and finally reaching close to Dueling Network performance using a normal Double DQN (DDQN).
I made the following changes to my code after closely examining the OpenAI baselines code:
Used PongFrameskip-v4 ...
3
Does the opponent's turn affect the calculated rewards?
Yes, in general it can. Obvious case, in a two player game where the opponent could win or lose on their turn, but has other options.
As far as I know, the reward should only be the result of the agent's action right?
In a well-defined MDP, the reward should be a stochastic function of the current ...
3
The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they noted later, the fact that DQN clips rewards anyways weakens this argument.
If you're going to use someone else's ready-made code for your prioritized experience ...
3
Including exogenous variables in your state representation certainly can be useful, as long as you expect them to be relevant information for determining the action to pick. So, state features are not only useful if you expect your agent (through application of actions) to have (partial) influence on those state variables; you just want the state variables ...
3
The loss function for DQN algorithm is
\begin{equation}
L(\theta_i) = \mathbb E_{s, a, r, s'} [(y - Q(s, a;\theta_i))^2]
\end{equation}
Like you said, we only take one action per timestep. We can only shift weights of the network that had the effect in calculating action value $Q(s, a)$ for that particular action that we took. For that action, variable $y$ ...
3
For the programming part I suggest this YouTube channel by Phil Tabor (he also has a website: neuralnet.ai. I found his videos really useful while I was attending reinforcement learning classes at the uni. He covers basic algorithms like value iteration and policy iteration and also more advanced like deep q learning, covering all main python libraries (...
3
Dueling-DQN has different network architecture comparing to vanilla DQN, so I don't think your version will work as well as the Dueling architecture.
From Wang et al., 2016, Dueling Network Architectures for Deep Reinforcement Learning
On the other hand, since we only have the target Q-value, separating the Q-value into state value and advantage result ...
3
Whats does the target Q-values represent?
In a DQN, which uses off-policy learning, they represent a refined estimate for the expected future reward from taking an action $a$ in state $s$, and from that point on following a target policy. The target policy in Q learning is based on always taking the maximising action in each state, according to current ...
3
The deep Q-learning (DQL) algorithm is really similar to the tabular Q-learning algorithm. I think that both algorithms are actually quite simple, at least, if you look at their pseudocode, which isn't longer than 10-20 lines.
Here's a screenshot of the pseudocode of DQL (from the original paper) that highlights the Q target.
Here's the screenshot of Q-...
2
I want to maximize the profit inside a trading day and avoid to place the pair (limit buy order, limit sell order) if the profit on that transaction is less than 100$. Be aware that I thought using the "Profit & Loss" as the reward.
To me this implies that your profit per transaction is not the true reward function that you should be using. You don't ...
2
Q learning predicts the action value, $q(s, a)$ for taking action $a$ in state $s$. The action value is usually the discounted sum of all future rewards. In general it can take any scalar value.
DQN uses a neural network to approximate $q(s, a)$. Although you might use this to select an action (thus think of the problem as a classification), the NN has to ...
2
$ \gamma $ goes up to 1, but cannot be greater than or equal to 1 (this would make the discounted reward infinite).
The discount factor $ \gamma $ determines the importance of future rewards. A factor of 0 will make the agent "myopic" (or short-sighted) by only considering current rewards, while a factor approaching 1 will make it strive for a long-term ...
2
After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were consecutives.
After redoing my experience selection method, my DQN is actually working and has reached about +200 points after 400000 experiences (about 500 episodes; ...
2
I have found some clues in Maei's thesis (2011): “Gradient Temporal-Difference Learning Algorithms.”
According to the thesis:
GTD2 is a method that minimizes the projected Bellman error (MSPBE).
GTD2 is convergent in non-linear function approximation case (and off-policy).
GTD2 converges to a TD-fixed point (same point as semi-gradient TD).
GTD2 is slower ...
2
Yes, you are exactly right. It is basically an arbitrary choice, although you should consider the reasonable numerical ranges of your activation functions if you decide to go beyond the values +/- 1. You can also have a think about whether you want to add a small reward for the agent reaching states that are near the goal, if you have an environment where ...
2
In Reinforcement Learning (RL), a reward function is part of the problem definition and should:
Be based primarily on the goals of the agent.
Take into account any combination of starting state $s$, action taken $a$, resulting state $s'$ and/or a random amount (a constant amount is just a random amount with a fixed value having probability 1). You should ...
2
In my humble opinion, it seems like it is important to have them separated, if having a certain card can influence the result in some way that is not its prime value, instead of not only using the sum. But it depends on the game and its rules. For example:
If having 5 cards of hearts in the set of 15 cards makes you win the game, then if you only represent ...
2
Your problem is not that the environment is stochastic or dynamic. In fact you are using the terms slightly incorrectly. These terms do not usually refer to the fact that starting state can differ or goal locations can move episode-by-episode. They typically refer to behaviour of state transitions.
Although in your case you could view the initial state as ...
2
I am not an expert in this area. But I believe that the word "Deterministic" is for "Policy" in the "Deterministic Policy" Gradient. It does not mean deterministic environment.
Stochastic policy: Probabilistic(random) action choice for a given state.
Deterministic policy: one action is chosen for a given state.
Deterministic Policy Gradient algorithm ...
2
The initial state can vary during in both training and use, and how you decide to do this makes very little difference to Q-learning. The important factor is whether all state/action pairs relevant to optimal behaviour can be reached. As there is already randomness in any exploring policy, and in many environments as part of state transitions and reward ...
2
Just as the paper says
$$L_i(\theta_i)= E_{(s,a)\sim p}[(y_i-Q(s,a;\theta_i))^2]$$
where
$$y_i = E_{s' \sim \mathcal{E}}[r+\gamma \max_{a'}Q(s',a';\theta_{i+1})\mid s,a]$$
Then in the Background section of the paper, it says
Differentiating the loss function with respect to the weights we arrive at the following gradient:
$$\nabla_{\theta_i} L_i(\...
Only top voted, non community-wiki answers of a minimum length are eligible
