New answers tagged reinforcement-learning
3
There are many algorithms that are not based on finding a value function. These are called policy gradients. These methods attempt to map states to actions through a neural network. They learn the optimal policy directly, not through a value function.
The important part of the image is when Model-Free RL splits into Policy Optimization and Q-Learning. ...
1
In the PDF of the original paper for UCB1 you linked, in page 242-243 the authors proves why non-optimal machines get played much less (in fact, logarithmically less) than the optimal ones. $c$ decides whether they indeed would, and $c=\sqrt{2}$ is the minimum choice of $c$.
We want to show that the number of runs for non-optimal machines ($n_i$, for non-...
0
I am confused. For the initial $Q$-values, you generate one for each possible row $(1, 0, 0), (0,0,0), \ldots$ so you would have 4 states.
However, from the first paragraph it seems that the states themselves are matrices (one row for each user), so the state space is a set of such matrices.
That means that your $Q$-table should have a row for each possible ...
1
It is not clear form your question, how you use your replay buffer. Basically, you have to store all states/reward tuples and train your agent on the entire buffer.
Moreover, you should give the agent time to explore (all) states of the world. But if you want to speed up training, you can try to implement importance sampling
1
The reward function belongs the the environment and it is the only way the agent can explore the world given a state.
If we want agent to do something specific, we must provide rewards to it in such a way that it will achieve our goals. It is thus very important that the reward function accurately indicates the exact behaviour.
Depending on your goal you can ...
1
The reward function is up to you when you set the goals for the agent.
If the goal is to score as highly as possible, before ending the game, then use the score. You may want to scale the score down if you are using neural networks, to prevent needing to handle very large error values in early phases of learning.
If the goal is to win the game, and you do ...
0
You can modify the environments and create custom models using Ray RLlib: https://docs.ray.io/en/latest/rllib.html
0
Since you are looking at a single iteration and expect a meaningful change my guess is that you aren't training for long enough. Q-learning can take very long, for many environments it takes millions of iterations.
0
You could use Ray RLlib. It has support for parallel environments, even over multiple GPUs and compute nodes.
1
The state space is certainly continuous, assuming that you can somehow feed that AI exact coordinates. You may have to resort to CNNs if you do not have access to this information. For the action space, you should consider how the game actually plays. Since you use a mouse to simply show the direction, you could use (x,y) positions of the mouse as an action, ...
0
I was thinking this strategy may work.
So, Q-learning takes vector input as state representation let's say your vector has n dimensions i.e. [$n_0$, $n_1$, $n_2$,..., $n_{n-1}$]
Now, from my interpretation you want to populate a matrix with 0 and 1's given the state vector but action-space has a high complexity e.g. an 8*8 matrix has 64 cells i.e. $2^{64}$ ...
3
I am using the convention of uppercase $X$ for random variable and lowercase $x$ for an individual observation. It is possible your source material did not do this, which might be causing your confusion. However, it is the convention used in Sutton & Barto's Reinforcement Learning: An Introduction.
What I didn't understand what is 𝑋 here. i.e., what is ...
1
Yes, this method of training a model is commonly known as online learning and specific learning algorithms have been designed for this purpose, such as, Stochastic Gradient Descent(SGD). As opposed to Batch Gradient descent, which computes gradients over the entire training set at each step, the SGD algorithm computes gradients for individual samples and ...
0
In the paper that you cite, Inverse Reward Design (2017), the authors actually define what they mean by "proxy reward function".
We formalize this in a probabilistic model that relates the proxy (designed) reward to the true reward
So, the proxy reward function is the reward function designed by the human, which may not necessarily be the reward ...
4
A typical and practical way to measure the convergence to some solution (so not necessarily the optimal one!) of any numerical iterative algorithm (such as RL algorithms) is to check if the current solution has not changed (much) with respect to the previous one. In your case, the solutions are value functions, so you could check if your algorithm has ...
1
Can my loss function be evaluating the model until it dies? 1/survival time could be the loss value to be minimized by gradient descent.
In order to use backpropagation and gradient descent, you have to relate the loss function directly to the output of the neural network. Your proposed loss function is too indirect, it is not possible to turn it directly ...
0
We don't need multiple environments.
On-policy algorithms require that new training samples are collected with the newest policy, so we can't use an experience buffer. However we can use the newest policy to collect multiple samples, even over multiple epochs, before updating the weights. This update can be a batch update.
1
"Will a neural network adapt to that ?"
No.
The big functional difference between human mind and neural networks : human mind learns by itself, a NN not.
If we call NN the net with its layers, weights, ... this is a static system, unable to learn anything new. The back-propagation algorithm that made intelligent the NN runs outside the NN itself, ...
1
I see some issues in your code of the environment. Firstly, and probably most importantly, you should not be incrementing the reward. In your code, every time the agent hits $t=475$ for example, the reward given by the environment increases by 1. So if the agent oscillates between $t=450$ and $t=475$, at each timestep the environment gives a greater and ...
1
The behaviour when playing against "cheats" depends on how the agent has been trained, and how different the game becomes from the training scenarios. It will also depend on how much of the agent's behaviour is driven by training, and how much by just-in-time planning.
In general, unless game playing bots are written specifically to detect or cope ...
4
In reinforcement learning (RL), there are some agents that need to know the state transition probabilities, and other agents that do not need to know. In addition, some agents may need to be able to sample the results of taking an action somehow, but do not strictly need to have access to the probability matrix. This might be the case if the agent is allowed ...
1
You might be able to glean what you want from Chapter 13 or Sutton & Barto's Reinforcement Learning: An Introduction, which deals with policy gradient algorithms, and includes pseudocode for a variety of agents based on linear approximation using softmax regression. From your description, you appear to be using - or should consider - softmax regression ...
1
To add to nbro's answer, I'd say also that much of the time the distance measure isn't simply a design decision, rather it comes up naturally from the model of the problem. For instance, minimizing the KL divergence between your policy and the softmax of the Q values at a given state is equivalent to policy optimization where the optimality at a given state ...
0
Our recent work solves this problem by using the idea of a forward-looking actor.
We use a neural network to forecast the next state given the current state and current action. Then plug it into the actor training with considering the value of future states. We use our idea on TD3 and make a new algorithm TD3-FORK, which solves this problem with as few as ...
1
We assume that our MDP is ergodic. Loosely speaking, this means that wherever the MDP starts (i.e. no matter which state we start in) or any actions the agent takes early on can only have a limited effect on the MDP and in the limit (as $t \rightarrow \infty$) the expectation of being in a given state depends only on the policy $\pi$ and the transition ...
2
$\mu(s)$ is not in equation (9.4) because we are assuming that the examples by which we update our parameter $w$, i.e. the frequency of which we will observe the states during online training, is the same. That is, it is a constant with respect to $w$ and since we are differentiating it can be somewhat disregarded as a constant of proportionality -- it ...
1
I did not read those two specified linked/cited papers and I am not currently familiar with the total variation distance, but I think I can answer some of your questions, given that I am reasonably familiar with the KL divergence.
When you compute the $D_{KL}$ between two polices, what does that tell you about them
The KL divergence is a measure of "...
1
In toy problems like the Short Corridor task, you can choose the state representation to explore a key property, such as the ability of a particular method to solve it. Often this is done to extremes and heavily simplified.
That is what is going on here. The state space that the agent is allowed to use is made highly degenerate with respect to the problem. ...
2
You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely uninformative.
This is why a stochastic policy is what is needed. This is common for POMDPs, whereas for regular MDPs we can always find a deterministic policy ...
0
As a supplement to nbro's nice answer, I think a major difference between RL and optimal control lies in the motivation behind the problem you're solving. As has been pointed out by comments and answers here (as well as the OP), the line between RL and optimal control can be quite blurry.
Consider the LQG algorithm (linear quadratic gaussian) which is ...
2
Section 5.2 Error Decomposition of the book Understanding Machine Learning: From Theory to Algorithms (2014) gives a description of the approximation error and estimation error in the context of empirical risk minimization (ERM), so in the context of learning theory. I will just summarise their definition. If you want to know more about these topics, I ...
1
Firstly, note that the Gaussian policies you describe are not equivalent to $\epsilon$-greedy, mainly for this reason: for a fixed policy, the policy's variance in the Gaussian case depends on the state, while in the $\epsilon$-greedy case it does not. Right off the bat, the Gaussian policy should achieve less regret than $\epsilon$-greedy.
Other approaches ...
2
How much the $Q$-values change does not depend on the value of $\epsilon$, rather the value of $\epsilon$ dictates how likely you are to take a random action and thus take an action that could give rise to a large TD error -- that is a large difference between the returns you expected from taking this action as to what you actually observed. How much the $Q$-...
2
This update rule can still be applied in the continuous domain.
As pointed out in the comments, suppose we are parameterising our policy using a Gaussian distribution, where our neural networks take as input the state we are in and output the parameters of a Gaussian distribution, the mean and the standard deviation which we will denote as $\mu(s, \theta)$ ...
0
This is more meant like a comment to the previous answer. I also originally thought that
$$ \nabla_{\theta}\log \pi_{\theta}(f_{\theta}(\varepsilon, s)\mid s) = \nabla_{a}\log\pi_{\theta}(a\mid s)\vert_{a=f_{\theta}(\varepsilon,s)}\nabla_{\theta}f_{\theta}(\varepsilon, s), $$ instead of
$$ \nabla_{\theta}\log \pi_{\theta}(f_{\theta}(\varepsilon, s)\mid s) = \...
0
I think this question is hinting at the problem of choosing an exploration strategy.
The simplest strategy is to use the so called epsilon-greedy strategy (or $\epsilon$-greedy). This means that you select an action at random $x$ percent of the times that an agent has to select an action. The other times, the agent takes the action that its current policy ...
0
I think it makes sense to use a Conv2D net for evaluating each position where you have different input channels for each figure type on the board. For example one channel for pawns:
an 8x8 matrix with 1's where there are white pawns, and -1's where there are black pawns. the rest should be 0.
Also input channels for bishops, knights etc... and then ...
1
It is not 100% clear, but this seems like an instance of catastrophic forgetting. This is something that often impacts reinforcement learning.
I have answered a very similar question on Data Science stack exchange, and reproduce the same answer here.
This is called "catastrophic forgetting" and can be a serious problem in many RL scenarios.
If you ...
2
In short, you don't regret your bad luck that you could do nothing about, you regret your bad choices that you could have done something about if only you knew.
The point of regret as a metric therefore is to compare your choices with the ideal choices. This makes sense in MABs, because although the primary goal is to gain the most reward, the learning part ...
4
Q-learning is said to be "model-free". Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken?
No. That's not why Q-learning is model-free. Q-learning assumes that the underlying environment (FrozenLake or MountainCar, for example) can be modelled as a Markov decision ...
Top 50 recent answers are included
