Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
1
vote
0answers
38 views

Dealing with input to recurrent net with changing dimensions

I have a problem in which the dimensions of the input are increasing in row and column at each timestep. What method for preprocessing could be done or are there any architectures used for solving ...
6
votes
1answer
4k views

How to implement exploration function and learning rate in Q Learning

I'm trying to implement Q-learning (state-based representation and no neural / deep stuff) but I'm having a hard time getting it to learn anything. I believe my issue is with the exploration function ...
2
votes
1answer
66 views

Programming a bandit to optimize donations

I'm developing a multi-armed bandit which learns the best information to display to persuade someone to donate to charity. Suppose I have treatments A, B, C, D (which are each one paragraph of text). ...
20
votes
2answers
9k views

What is sample efficiency, and how can importance sampling be used to achieve it?

For instance, the title of this paper reads: "Sample Efficient Actor-Critic with Experience Replay". What is sample efficiency, and how can importance sampling be used to achieve it?
1
vote
1answer
168 views

Agent in toy environment only learns to act optimally with small discount factors

I have tried several environment libraries like OpenAI gym/gridworld but now I am trying to create a toy environment for experimentation. The environment I've created is as follows: State: grid with ...
4
votes
2answers
90 views

Move blocks to create a designed surface

I am new to machine learning and AI, so forgive me if this is obvious. I was talking with a friend on how to solve this problem, and neither of us could figure out how to do it. Say I have a grid ...
4
votes
2answers
1k views

What layers to use in a Neural Network for card game

I am currently writing an engine to play a card game and I would like for an ANN to learn how to play the game. The game is currently playable, and I believe for this game a deep-recurrent-Q-network ...
5
votes
1answer
220 views

What is a weighted average in a non-stationary k-armed bandit problem?

In the book Reinforcement Learning: An Introduction (page 25), by Richard S. Sutton and Andrew G. Barto, there is a discussion of the k-armed bandit problem, where the expected reward from the bandits ...
2
votes
1answer
647 views

RL agent's view of state transitions

The above environment is DeepTraffic Now consider this situation in the above environment, the Red car (we control it with our RL agent) is on the extreme right lane. During the exploration phase, ...
0
votes
1answer
51 views

How to classify this environment?

Consider an environment, where an agent intends to move from cell "A" to cell "G", avoiding obstacles (cells marked with shading). The agent can move forward, rotate 90º to the ...
1
vote
1answer
515 views

What is the relation between back-propagation and reinforcement learning?

What is the relation between back-propagation and reinforcement learning?
3
votes
1answer
484 views

Which Reinforcement Learning algorithms are efficient for episodic problems?

I have some episodic datasets extracted from a turn-based RTS game in which the current actions leading to the next state doesn’t determine the final solution/outcome of the episode. The learning is ...
2
votes
0answers
89 views

Which features and algorithm could optimize this air-conditioner problem?

Imagine we have 2 air conditioner systems (AA) and 2 "free cooling" systems which mix external and internal air (FC) in a closed box which always tends to warm up. For each system, we have to find ...
3
votes
1answer
269 views

Can we teach machine to tie shoe lace?

Is it possible with any of machine learning methods to train machine to tie shoe lace? If possible how data should be interpreted for the training? If we are using reinforcement learning, how will it ...
2
votes
1answer
56 views

What is uncontrolled logging policy?

I am reading Learning from Logged Implicit Exploration Data It says Formally, given a dataset of the form S = (x, a, r_a)* generated by the interaction of an uncontrolled logging policy What is ...
1
vote
0answers
46 views

Regarding Tensorflow: How to Avoid Duplicate Use of Scope/Variable_names

I am trying to train Chess data through CNN. To proceed reinforcement learning, I had divided into two - "current network" and "reinforcement network". For each checkpoint file stored in different ...
10
votes
1answer
2k views

Do off-policy policy gradient methods exist?

Do off-policy policy gradient methods exist? I know that policy gradient methods themselves using the policy function for sampling rollouts. But can't we easily have a model for sampling from the ...
5
votes
2answers
1k views

Is Q-learning a type of model-based RL?

Model-based RL creates a model of the transition function. Tabular Q-Learning does this iteratively (without directly optimizing for the transition function). So, does this make tabular Q-learning a ...
6
votes
1answer
223 views

Why do Bellman equations indirectly create a policy?

I was watching a lecture on policy gradients and Bellman equations. And they say that a Bellman equation indirectly creates a policy, while the policy gradient directly learns a policy. Why is this?
7
votes
2answers
2k views

How do I create an AI for a two-players board game?

Goal I want to create an artificial intelligence to compete against other players in a board game. Game explanation I have a board game similar to 'snakes and ladders'. You have to get to a final ...
5
votes
1answer
1k views

Why is the target $r + \gamma \max_{a'} Q(s', a'; \theta_i^-)$ in the loss function of the DQN architecture?

In the paper Human-level control through deep reinforcement learning, the DQN architecture is presented, where the loss function is as follows $$ L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \...
2
votes
1answer
90 views

Why is the access to the dynamics model unrealistic in Q-Learning?

Pieter Abbeel says that having access to the dynamics model, $P(s' \mid s,a)$, is unrealistic because it assumes we know the probability that we will reach all future states. I don't understand how ...
8
votes
1answer
415 views

What are the different approaches used in Machine Learning?

There seem to be so many sub-fields, so I'm interested in getting a better understanding of the approaches. I'm looking for information on a single framework per answer, in order to allow for ...
2
votes
1answer
220 views

Is it expected that during self-play reinforcement learning that player 1 or player 2 wins the majority of games?

I'm testing various learning rates and neural network configurations. I'm testing over 10000 games, with the first 2000 having random starting moves and general randomness throughout of about 20%, i.e....
3
votes
1answer
138 views

Why are neural networks always trained “by themselves”?

In the current rush of artificial intelligence research, fueled by NN, independent of the paper I choose, the NN are always trained by themselves. Sure, there are architectures that combine CNN and ...
7
votes
2answers
5k views

How to combine backpropagation in neural nets and reinforcement learning?

I have followed a course on machine learning, where we learned about the gradient descent (GD) and back-propagation (BP) algorithms, which can be used to update the weights of neural networks, and ...
1
vote
0answers
110 views

When do you back-propagate errors through a neural network when using TD($\lambda$)?

I have a neural network that I'm want to use to self-play Connect Four. The neural network receives the board state and is to provide an estimate of the state's value. I would then, for each move, ...
10
votes
1answer
2k views

How could I use reinforcement learning to solve a chess-like board game?

I invented a chess-like board game. I built an engine so that it can play autonomously. The engine is basically a decision tree. It's composed by: A search function that at each node finds all ...
58
votes
6answers
48k views

What's the difference between model-free and model-based reinforcement learning?

What's the difference between model-free and model-based reinforcement learning? It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In ...
7
votes
2answers
814 views

Where to publish a first article in Deep Reinforcement Learning?

What would be examples of journals that are good for a first publication in the field of Deep Reinforcement Learning? I am in the process of writing about the research results of DQN-related ...
3
votes
1answer
122 views

Programming an inference AI that computes the best outcomes like a quantum computer

I bought an Intel Movidius Neural Compute stick a few weeks ago. Even though I can use it with the examples, I want to actually use it for something! The documentation is messy, and hard to work ...
1
vote
0answers
66 views

Handling varied-size input with fixed-input network

I'm running A3C (Asynchronous Actor-Critic Agents) to learn a game where an agent needs to catch 3 rewards. The input of my network, among other things, is the relative position of the 3 rewards ...
1
vote
0answers
173 views

Help with implementing Q-learning for a feedfoward network playing a video game

I want to train a feedforward neural network to play a video game called Puyo Puyo 2, using reinforcement learning. More specifically, I'm trying Q-learning but I'm open to better alternatives. In ...
2
votes
1answer
258 views

Does eligibility traces and epsilon-greedy do the same task in different ways?

I understand that, in Reinforcement Learning algorithms, such as Q-learning, to prevent selecting the actions with greatest q-values too fast and allow for exploration, we use eligibility traces. Here ...
1
vote
1answer
207 views

What is $I$ in the noise described in the paper “Parameter Space Noise for Exploration”?

In the paper Parameter Space Noise for Exploration, the authors describe the noise that they add to the parameter vector as: $$ \tilde{\theta} = \theta + \mathcal{N}(0, \sigma^2I) $$ is $I$ simply ...
1
vote
1answer
570 views

Reinforcement learning for robotic motion planning - Problem statement ideas

I am a first-semester grad student in Robotics and have taken a course on machine learning for robotics. I am completely new to machine learning. I am to select and execute a problem statement on my ...
2
votes
0answers
173 views

Is iLQG a good algorithm for model-based planning with simple environments?

In their work Continuous Deep Q-Learning with Model-based Acceleration, the author demonstrate great results of applying Imagination Rollouts for model-based acceleration of learning. They test their ...
5
votes
2answers
342 views

Inconsistency in TD-Leaf algorithm in KnightCap chess engine

Notice that, in the following formula, at the very right, the term multiplied with $\lambda$ is $d_i$ $$ w := w + \alpha \sum_{i=1}^{N-1} \nabla r(x_i^l, w) \Big \lfloor \sum_{j=i}^{N-1} \lambda^{j-i}...
14
votes
1answer
7k views

How can policy gradients be applied in the case of multiple continuous actions?

Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms. When using a single continuous action, normally, you would use some ...
4
votes
1answer
1k views

Is it necessary to clear the replay memory regularly in a DQN when an agent plays against itself?

I studied the article "Demystifying Deep Reinforcement Learning" extensively during the last days, while trying to implement the proposed algorithms myself. My goal is to have an agent learn by ...
4
votes
1answer
130 views

How can I design a hierarchy of agents each of which with different goals?

I read some light material earlier about the possibility of building a hierarchy of agents, where the agents at the leaves solve primitive tasks while higher-level agents are optimized for ...
4
votes
1answer
1k views

Is a decision tree less suitable for incremental learning than e.g. a neural net?

I can recall that a professor once said that decision trees are not good for incremental learning, as they have to be rebuilt from the ground up if new training examples arrive. Is this basically ...
4
votes
1answer
767 views

Traveling salesman problem variant: which algorithm to choose?

I have an industrial problem which I'm trying to cast as a Traveling Salesman problem (TSP) in 3D euclidian space. There are physical limitations which implies that some subpaths may or may not be ...
3
votes
1answer
241 views

What type of reinforcement learning can I do restricted to ~200MB on an average smartphone?

This concerns a set of finite, non-trivial, combinatorial games [M] in the form of an app. A sample game can be found here. Because this is a mass market product, we can't take up too much space, ...
5
votes
1answer
182 views

Did the Facebook robots both want everything but the balls?

According to this article, two Facebook ai's had the following "creepy" negotiation over a transaction: Bob: i can i i everything else . . . . . . . . . . . . . . Alice: balls have zero to ...
6
votes
2answers
946 views

How should I handle action selection in the terminal state when implementing SARSA?

I recently started learning about reinforcement learning. Currently, I am trying to implement the SARSA algorithm. However, I do not know how to deal with $Q(s', a')$, when $s'$ is the terminal state. ...
6
votes
1answer
1k views

Can reinforcement learning algorithms be applied to computer vision problems?

Can reinforcement learning algorithms be applied to computer vision problems? If yes, what are some examples of these applications?
4
votes
1answer
321 views

What's stopping Cepheus from generalizing to full poker games?

Cepheus is an artificial intelligence designed to play Texas Hold'em. By playing against itself and learning where it could have done better, it became very good at the game. Slate Star Codex comments:...
1
vote
1answer
195 views

Markov Model for a Traffic Intersection

I need some help in developing a Markov Model for a crossroads there is no one way road and i am assuming at this time that traffic is only allowed to go straight no turns are allowed. There are 4 ...
1
vote
2answers
2k views

Q Learning Algorithm not converging

I am trying to run Deep Q-learning algorithm on a game which I made in Python using pygame library. The algorithm accepts the game screen (4 frames) as input to neural network which used as the ...