Questions tagged [monte-carlo]

For questions related to the Monte Carlo methods in reinforcement learning and other AI sub-fields. ("Monte Carlo" refers to random sampling of the search space.)

Filter by
Sorted by
Tagged with
2
votes
1answer
62 views

Monte Carlo learning for Reinforcement learning

When you train a model using Monte Carlo-based learning the state and action taken at each step is recorded, and then at some point an end state is reached and the agent receives some reward - what do ...
2
votes
0answers
42 views

How to show Monte Carlo methods converge to an estimate which minimizes mean squared error?

In chapter six of Sutton and Barto (p.128), they claim Monte Carlo methods converge to an estimate minimizing the mean squared error. How can this be shown formally? Bump
1
vote
1answer
83 views

How is Monte Carlo different from model-based methods?

I was going through an article where it is mentioned: The Monte-Carlo methods require only knowledge base (history/past experiences)—sample sequences of (states, actions and rewards) from the ...
1
vote
1answer
50 views

Why does GLIE+MC Control Algorithm use a single episode of Monte Carlo evaluation?

GLIE+MC control Algorithm: My question is why does this algorithm use only a single Monte Carlo episode (during PE step) to compute the $Q(s,a)$? In my understanding this has the following drawbacks: ...
2
votes
2answers
61 views

How to stop evaluation phase in reinforcement learning with epsilon-greedy Monte Carlo agent?

I have implemented an epsilon-greedy Monte Carlo reinforcement learning agent like suggested in Sutton and Barto's RL book (page 101). As far as I understood epsilon-greedy agents so far, the ...
1
vote
0answers
24 views

How do I know if the assumption of a static environment is made?

An important property of a reinforcement learning problem is whether the environment of the agent is static, which means that nothing changes if the agent remains inactive. Different learning methods ...
3
votes
1answer
64 views

RL Monte Carlo update

In Sutton & Barto's Reinforcement Learning: An Introduction, in page 83 (101 of the pdf), there is a description of first-visit MC Control. In the phase where they update ...
2
votes
1answer
64 views

What is the relation between Monte Carlo and model-free algorithms?

Monte Carlo (MC) methods are methods that use some form of randomness or sampling. For example, we can use an MC method to approximate the area of a circle inside a square: we generate random 2D ...
2
votes
0answers
10 views

How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?

In section 3 of this paper the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator. I have read it a number of times and ...
0
votes
1answer
57 views

Monte-Carlo, every-visit gridworld, exploring starts, python code gets stuck in foreverloop in episode generation

I've been trying to implement policy improvement for Q(s,a) function as per Sutton&Barto reinforcement learning book. The original algorithm with first-visit MonteCarlo is pictured below. I ...
1
vote
0answers
68 views

Difficulty understanding Monte Carlo policy evaluation (state-value) for gridworld

I've been trying to read Sutton & Barto book chapter 5.1, but I'm still a bit confused about the procedure of using Monte Carlo policy evaluation (p.92), and now I just cant proceed anymore coding ...
2
votes
1answer
76 views

Why is Monte Carlo used as the tree search algorithm for AlphaGo?

Could a better algorithm other than Monte Carlo be used for the AlphaGo computer? Why didn't the DeepMind team think of choosing another kind of algorithm rather than spending time on their neural ...
1
vote
1answer
51 views

Similarities and differences between UCT algorithms in (i), (ii), (iii) and (iv)?

I am trying to understand the similarities and differences between: (i) the UCT algorithm in Kocsis and Szepesvári (2006); (ii) the UCT algorithm in Section 3.3 of Browne et al (2012); (iii) the MCTS ...
5
votes
1answer
55 views

Why didn't champion of the Go game manage to win the last game against AlphaGo, after winning the 4th one?

In the documentary about the match, it is said that after losing the 4th game, AlphaGo came back stronger and started to play in a weird way (not human-like) and it was pretty impossible to be beaten. ...
3
votes
1answer
975 views

What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?

I came across this 2 algorithms but I cannot understand the difference between these 2 both in terms of implementation as well as intuitionally. So what difference does the second point in both the ...
3
votes
1answer
80 views

What are temporal-difference and Monte Carlo methods intuitively?

Intuitively, how do temporal-difference and Monte Carlo methods work in reinforcement learning? How can they be used to solve the reinforcement learning problem?
2
votes
1answer
96 views

Do we need the transition probability function when calculating the importance sampling ratio?

I am reading the book titled "Reinforcement Learning: An Introduction" (by Sutton and Barto). I am at chapter 5, which is about Monte Carlo methods, but now I am quite confused. There is one thing I ...
4
votes
1answer
767 views

Why is GLIE Monte-Carlo control an on-policy control?

In slide 16 of his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control. But why is it an on-policy control? The sampling follows a policy $\pi$ while ...