Questions tagged [model-based-methods]

For questions about model-based reinforcement learning methods (or algorithms). An example of a model-based algorithm is Dyna-Q, which estimates a model of the environment (i.e. the transition function of the associated Markov decision process).

Filter by
Sorted by
Tagged with
1
vote
1answer
62 views

Model-based RL for time series data

I have time-series data. When I take an action, it impacts the next state, because my action directly determines the next state, but it is not known what the impact is. To be concrete: I have $X(t)$ ...
0
votes
1answer
49 views

Can you use machine learning for binary data?

I am totally new to artificial intelligence and neural networks and have a broad question that I hope is appropriate to ask here. I am an ecologist working in animal movement and I want to use AI to ...
1
vote
0answers
59 views

What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?

In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods. AlphaGo Deepmind RL model has beaten the best Go ...
0
votes
1answer
38 views

When we have multiple traces, do we average over traces or the total number of times we have visited that state?

I am confused about the workings of the first- and every-visit MC. My first question is, when we have multiple traces, do we average over traces or the total number of times we have visited that state?...
0
votes
0answers
15 views

Using an LSTM for model-based RL in a POMDP

I am trying to set up an experiment where an agent is exploring an n x n gridworld environment, of which the agent can see some fraction at any given time step. I'd like the agent to build up some ...
1
vote
1answer
83 views

Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?

Here is David Silver's lecture on that. Look at 9:30 to 10:30. He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $Q$ is used. But ...
4
votes
1answer
162 views

Is the state transition matrix known to the agents in a Markov decision processes?

The question is more or less in the title. A Markov decision process consists of a state space, a set of actions, the transition probabilities and the reward function. If I now take an agent's point ...
2
votes
1answer
50 views

Are linear approximators better suited to some tasks compared to complex neural net functions?

Model based RL attempts to learn a function $f(s_{t+1}|s_t, a_t)$ representing the environment transitions, otherwise known as a model of the system. I see linear functions are still being used in ...
3
votes
2answers
107 views

Have agents that “dream” been explored in Reinforcement Learning?

I was reading this article about the question "Why do we dream?" in which the author discusses dreams as a form of rehearsal for future threats, and presents it as an evolutive advantage. My ...
1
vote
1answer
45 views

What is the expectation of an empirical model in model based RL?

In the paper - "Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems", on page 1083, on the 6th line from the bottom, the authors define ...
2
votes
1answer
83 views

Into which subcategories can reinforcement learning be divided?

In the course of a scientific work, I will discuss the different types of reinforcement learning. However, I have difficulties to find these different types. So, into which subcategories can ...
0
votes
0answers
44 views

Model Based rl and cross entropy method with nonlinear function approximators

Pseudo code for Cross entropy method according to youtube lecture 32:55 Initialize $\mu \in R^{d}, \sigma \in R^{d}$ iteration 1,2,... Collect n samples of $\theta_{i} \sim N(\mu,diag(\sigma))$ ...
3
votes
1answer
47 views

Why is learning $s'$ from $s,a$ a kernel density estimation problem but learning $r$ from $s,a$ is just regression?

In David Silver's 8th lecture he talks about model learning and says that learning $r$ from $s,a$ is a regression problem whereas learning $s'$ from $s,a$ is a kernel density estimation. His ...
1
vote
0answers
50 views

Using a model-based method to build an accurate day trading environment model

There are several different angles we can classify Reinforcement Learning methods from. We can distinguish three main aspects : Value-based and policy-based On-policy and off-policy Model-free and ...
5
votes
3answers
77 views

Isn't a simulation a great model for model-based reinforcement learning?

Most reinforcement learning agents are trained in simulated environments. The goal is to maximize performance in (often) the same environment, preferably with a minimum amount of interactions. Having ...
3
votes
1answer
65 views

How can the policy iteration algorithm be model-free if it uses the transition probabilities?

I'm actually trying to understand the policy iteration in the context of RL. I read an article presenting it and, at some point, a pseudo-code of the algorithm is given : What I can't understand is ...
4
votes
1answer
131 views

Is the minimax algorithm model-based?

Trying to get my head around model-free and model-based algorithms in RL. In my research, I've seen the search trees created via the minimax algorithm. I presume these trees can only be created with a ...
0
votes
0answers
25 views

Concrete examples of models and policies in Tic Tac Toe environment

I'm having difficulty picturing how models and policies are represented. Could someone describe how they would look in the context/environment of a game of Tic Tac Toe? For example, "In Tic Tac Toe, ...
2
votes
0answers
53 views

Correlating two models to predict the output of one that corresponds to an output of the other

I am currently working on a problem and now got stuck to implement one of it's steps. This is a simple attempt to explain what I am currently facing, which is something that I am aiming to implement ...
3
votes
1answer
447 views

Why are model-based methods more sample efficient than model-free methods?

Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two ...
2
votes
1answer
180 views

How is Monte Carlo different from model-based methods?

I was going through an article where it is mentioned: The Monte-Carlo methods require only knowledge base (history/past experiences)—sample sequences of (states, actions and rewards) from the ...
1
vote
0answers
24 views

Architecture and Use of Different Algorithms for Health Goal Feedback

I wanted to get some opinions from the community for a certain problem that I will be approaching. The problem is to provide feedback to a user based on a image of the upper male torso. The image ...
5
votes
0answers
61 views

Choosing Machine Learning Algorithm: Learning-Based Testing

This is my first project using machine learning so I'm looking for some guidance. I am extending a model-based testing (MBT) system to a learning-based testing system by integrating a machine learning ...
2
votes
1answer
87 views

Is there any grid world dataset or generator for reinforcement learning?

I would like to start programming a multi task reinforcement learning model. For this, I need not just one maze or grid world (or just model-based), but many with different reward functions. So, I am ...
1
vote
2answers
252 views

How can we estimate the transition model and reward function?

In reinforcement learning (RL), there are model-based and model-free algorithms. In short, model-based algorithms use a transition model (e.g. a probability distribution) and the reward function, even ...
3
votes
1answer
115 views

How do temporal-difference and Monte Carlo methods work, if they do not have access to model?

In value iteration, we have a model of the environment's dynamics, i.e $p(s', r \mid s, a)$, which we use to update an estimate of the value function. In the case of temporal-difference and Monte ...
2
votes
0answers
66 views

Eligibility trace In Model-based Reinforcement Learning

In model-based reinforcement learning algorithms, the model of the environment is constructed to efficiently use samples, models such as Dyna, and Prioritize Sweeping. Moreover, eligibility trace ...
4
votes
2answers
1k views

Is Q-learning a type of model-based RL?

Model-based RL creates a model of the transition function. Tabular Q-Learning does this iteratively (without directly optimizing for the transition function). So, does this make tabular Q-learning a ...
50
votes
6answers
35k views

What's the difference between model-free and model-based reinforcement learning?

What's the difference between model-free and model-based reinforcement learning? It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In ...