# Questions tagged [monte-carlo-methods]

For questions related to the Monte Carlo methods in reinforcement learning and other AI sub-fields. ("Monte Carlo" refers to random sampling of the search space.)

83 questions
Filter by
Sorted by
Tagged with
1 vote
45 views

This is my understanding thus far about Monte Carlo method for approximating value function: Instead of using a recursive Bellman equations and knowledge of environment dynamics, Monte Carlo methods ...
1 vote
35 views

### Understanding the policy improvement theorem for Monte Carlo Control without Exploring Starts

I am currently studying the equations 5.2 in Reinforcement Learning An Introduction By Sutton and Barto on page 101. I want to comprehent the proof by a simple example: Having only one State with two ...
• 13
40 views

### Why is there the potential problem of "learning only from the tails of episodes" in off-policy MC control?

Sutton-Barto page 111, first paragraph (Off-policy Monte Carlo Control): A potential problem is that this method learns only from the tails of episodes, when all of the remaining actions in the ...
• 974
1 vote
274 views

### Confused about Monte Carlo first visit algorithm

I'm really confused about understanding the Monte Carlo first visit algorithm as presented in the Sutton & Barto's book in chapter V. Here is the pseudocode: The reason is that I'm trying to ...
• 204
64 views

### Finding a value at which TD(lambda) and Monte Carlo Are Similar

Let's say that I'm trying to find a value of lambda for which running a TD(lambda) method will produce the same results as a Monte Carlo method (within a small margin or error, of course). One of the ...
237 views

### How does Monte-Carlo Tree Search Compare to MCMC?

Monte-Carlo Tree Search was the method used for AlphaGo my understanding is: it would randomly search the state space of possible moves where the probability of choosing a move was proportional to the ...
• 434
285 views

### How does off-policy Monte Carlo weighted importance sampling bias converge to zero (Sutton & Barto Section 5.5)

On Section 5.5 (page 105) of Sutton & Barto's "Reinforcement Learning: An Introduction", they discuss the off-policy Monte Carlo method for learning the value function of a target policy ...
• 248
1 vote
237 views

• 225
251 views

### Comparison between TD(0) and MC ( or GAE )?

I'm getting started with DRL and have trouble distinguishing TD(0), MC, and GAE; and which scenarios one's better than others. Here is what I understand so far: TD(0): increment learning, can learn ...
314 views

### With Monte Carlo off-policy learning what do we correct by using importance sampling?

I do not understand the link of importance sampling to Monte Carlo off-policy learning. We estimate a value using sampling on whole episodes, and we take these values to construct the target policy. ...
615 views

### In MCTS, what to do if I do not want to simulate till the end of the game?

I'm trying to implement MCTS with UCT for a board game and I'm kinda stuck. The state space is quite large (3e15), and I'd like to compute a good move in less than 2 seconds. I already have MCTS ...
• 63
1 vote
1k views

### How exactly is Monte Carlo counterfactual regret minimization with external sampling implemented?

I have read many papers, such as this or this, explaining how external sampling works, but I still don't understand how the algorithm works. I understand you divide $Q$, which is the set of all ...
• 11
403 views

### What are the popular approaches to estimating the Q-function?

I need the q-value for my RL training, there are some approaches: Brute-force the action sequence (this won't work for long sequence) Use a classic algorithm to optimise and estimate (this ain't much ...
• 1,293
516 views

### How can I use Monte Carlo Dropout in a pre-trained CNN model?

In Monte Carlo Dropout (MCD), I know that I should enable dropout during training and testing, then get multiple predictions for the same input $x$ by performing multiple forward passes with $x$, then,...
• 31
1k views

### Why do we need importance sampling?

I was studying the off-policy policy improvement method. Then I encountered importance sampling. I completely understood the mathematics behind the calculation, but I am wondering what is the ...
1 vote
230 views

### When updating the state-action value in the Monte Carlo method, is the return the same for each state-action pair?

Referring to this post, in the following formula to update the state-action value $$Q(s,a) = Q(s,a) + \alpha (G − Q(s,a)),$$ is the value of $G$ (the return) the same for every state-action $(s,a)$ ...
669 views

### Unclear definition of a "leaf" and diverging UTC values in the Monte Carlo Tree Search

I have two questions regarding the Selection and Expansion steps in the Monte Carlo Tree Search Algorithm. In order to state the questions, I recall the algorithm that I believe is the one most ...
• 123
492 views

### Is my pseudocode titled "Monte Carlo Exploring Starts (with model)" correct?

Reinforcement Learning: An Introduction second edition, Richard S. Sutton and Andrew G. Barto: We made two unlikely assumptions above in order to easily obtain this guarantee of convergence for the ...
130 views

### When we have multiple traces, do we average over traces or the total number of times we have visited that state?

I am confused about the workings of the first- and every-visit MC. My first question is, when we have multiple traces, do we average over traces or the total number of times we have visited that state?...
514 views

### Suppose every-visit MC was used instead of first-visit MC on blackjack. Would you expect the results to be different?

This is a question from page 94 of Sutton and Barto's RL book 2020. I read in someone's compiled GitHub answers to this book's exercises their answer was: "No because each state in an episode of ...
• 805
218 views

### Is the expected value we sample in TD-learning action-value Q or state-value V?

Both MC and TD are model-free and they both follow a sample trajectory (in the case of TD, the trajectory is cut-short) to estimate the return (we basically are sampling Q values). Other than that, ...
• 155
1 vote
464 views

### Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?

Here is David Silver's lecture on that. Look at 9:30 to 10:30. He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $Q$ is used. But ...
• 155
320 views

1k views

### In what RL algorithm category is MiniMax?

Q-learning is a temporal-difference method and Monte Carlo tree search is a Monte Carlo method. In what category is MiniMax?
• 623
48 views

### Why do we update $W$ with $\frac{1}{\mu (A_t | S_t)}$ instead of $\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$ in off-policy Monte Carlo control?

I had the same question when I am reading the RL textbook from Sutton Bartol as posted here. Why do we update $W$ with $\frac{1}{\mu (A_t | S_t)}$ instead of $\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$?...
• 53
1 vote