5
A stochastic process has the Markov property if the probability distribution of future states conditioned on both the present and past states depends only on the present state or, more formally, the following equality holds.
$$
p(s_{t+1} \mid s_{t}, s_{t-1:1}) = p(s_{t+1} \mid s_{t}), \forall t
$$
The hidden Markov model (HMM) is an example of a model ...
3
Automated machine learning (AutoML) is an umbrella term that encompasses a collection of techniques (such as hyper-parameter optimization or automated feature engineering) to automate the design and application of machine learning algorithms and models.
Reinforcement learning (RL) is a sub-field of machine learning concerned with the task of making ...
3
I believe this is a pedagogical decision. Because the sentence occurs in the first chapter of the book, I think the authors are trying to avoid the objection that a neophyte might make: learning from random movements seems like it will cause you to learn strange behaviors.
Certainly, the statement is inaccurate. We need only reach page 26 to see a ...
2
So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?
Yes.
Why would you do that? To have the first layer focus on learning the state value independent of the selected action? How would that help?
Neural networks hidden ...
2
So, what is the purpose of the new index for $V$ in Chapter 7, and why is it more important at this particular chapter?
My guess would be that your intuition is correct, and that it's mostly introduced just to clarify exactly which "version" of our value function approximator is going to be used in any particular equation. In previous chapters, which ...
2
Dealing with a Non-Markovian process is unusual in Reinforcement Learning. Although some explicit attempts have been made, the most common approach when confronted with a non-Markovian environment is to try and make the agent's representation of it Markovian.
After reducing Agent's model of the dynamics to a Markovian process, rewards are assigned from the ...
2
RL can be used in the context of Neural Architecture Search (NAS), with is a form of automated ML. A model searches for an architecture that performs a given task. How well this task is performed guides how the architecture will be modified (improved) on the next pass. It works but is very computation-intensive (think hundreds of GPUs).
See for instance:
B....
2
Hi Seewoo Lee and welcome to our community!
The essence of your observation is that Sutton's version of REINFORCE is taking into consideration all of the trajectory to compute the returns while in the pytorch version only the future is taken into consideration, hence going in reverse to sum the future rewards and ignore the previous rewards. The consequence ...
2
Although you can frame your problem as a bandit problem or RL, it has other workable interpretations. Critical information from your comments is that:
Total reward is not a simple sum of all the results from 66 different machines. There are interactions between machines.
Total reward is deterministic.
This looks like a problem in combinatorial optimisation....
2
The double sampling problem is referenced in Chaper 11.5 Gradient Descent in the Bellman Error in Reinforcement Learning: An Introduction (2nd edition).
From the book, this is the full gradient descent (as opposed to semi-gradient descent) update rule for weights of an estimator that should converge to a minimal distance from the Bellman error:
$$w_{t+1} ...
1
You cannot do this:
$\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )]$
That is because $r(\tau )$ and $\bigtriangledown log \pi (\tau )$ are correlated by their dependence on $\tau$. In a simpler concrete example, if your expectation ...
1
logp seen in code is actually logit p which has this story behind:
Given a probability p, the corresponding odds are calculated as p / (1 – p). For example if p=0.75, the odds are 3 to 1: 0.75/0.25 = 3.
The logit function is simply the logarithm of the odds: logit(x) = log(x / (1 – x)).
Sigmoid near logp is like follows:
The inverse of the logit ...
1
You are missing that the expression
$$\sum_{s'} \eta(s')$$
is already a count of the expected length of an episode, and is used in the denominator to scale $\mu(s)$ such that $\sum_{s} \mu(s) = 1$
So the length of the episode is taken into account in the formula.
In practice you don't need to know $\mu(s)$, it can be left unresolved as a theoretical ...
1
Let's first assume that there is only one action so that $\pi(a|s) = 1$ for every state - action pair which simplifies the discussion.
Now let's consider a case with 100 time steps, 10 states and uniform distribution for starting state $s_0$ with $h(s_0) = 1$. The result would be
\begin{align}
\eta(s_0) &= 1 + \sum_{i = 0}^9 \eta(s_i) \cdot p(s_0|s_i) =\\...
1
In general, you can use a simulation to prepare and train a controller for a real world application. A good example of this being done for robotics is in the paper Autonomous helicopter flight via reinforcement learning where a Reinforcement Learning agent was trained on a model of helicopter dynamics before being used in reality. Often, as in this case, ...
1
They don't have acces to the original training or test dataset. Machine learning environments are build on the premise of a benign environment. The models are trained on real data (real inputs). When someone sends a made up input (fake input) it is very easy to fool the model.
This is used for example in image recognition. Imagine a fotograph of a panda. ...
1
In deep reinforcement learning for portfolio optimization, many researchers (Xiong at al. for example) use historical market data for model training. The resulting MDP dynamics is of course completely deterministic (if historical prices are used as states) and there's no real sequentiality involved.
Whilst I cannot comment on the specific financial model, I ...
1
There is a platform called Flow, sounds like its what you're looking for
https://github.com/flow-project/flow
1
In step 2 I need to decide for an initial estimate $\tilde{Q}_n$. Is it a decent option to use $\tilde{Q}_n=Q_{n-1}$?
Yes, this is a common choice. It's actually common to update the table for $\tilde{Q}$ in place, without any separate initialisation per step. The separate phases of estimation and policy improvement are easier to analyse for theoretical ...
1
From your description, it seems that you are implementing a version of an algorithm called REINFORCE. This algorithm belongs to a family called Policy Gradient methods, which directly optimizes the policy network $\pi(a_t|s_t)$ from rewards without ever worrying about estimating a value function. This type of algorithm is usually pretty slow and presents ...
1
Page 6 of the paper describes the exact reward functions, and why they were used:
Goals: Goals describe the desired position of the object (a box or a
puck depending on the task) with some fixed tolerance of $\epsilon$ i.e. $G = \mathcal{R}^3$
and $f_g(s) = [|g − s_{object}| ≤ \epsilon]$, where $s_{object}$ is the position of
the object in the ...
1
Is it possible for value-based methods to learn stochastic policies?
Yes, but only in a limited sense, due to the ways it is possible to generate stochastic policies from a value function. For instance, the simplest exploratory policy used by SARSA and Monte Carlo Control, $\epsilon$-greedy, is stochastic.
SARSA natually learns the optimal $\epsilon$-...
1
Unfortunately no, the way to go is track the total reward and see if it's increasing and converging eventually. Value loss isn't a useful metric as the loss can be 0 when the value network always predicts 0 and the agent doesn't collect any reward, meaning very poor behavior.
1
https://github.com/openai/retro
Current list of machines is
Atari
Atari2600 (via Stella)
TurboGrafx-16/PC Engine (via Mednafen/Beetle PCE Fast)
Game Boy/Game Boy Color (via gambatte)
Game Boy Advance (via mGBA)
Nintendo Entertainment System (via FCEUmm)
Super Nintendo Entertainment System (via Snes9x)
GameGear (via Genesis Plus GX)
Genesis/Mega Drive (via ...
Only top voted, non community-wiki answers of a minimum length are eligible
