Hot answers tagged

4

Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value V(s) function is estimated separately from the state-action value Q(s, a). A new quantity, the advantage of an action, can then be defined as A(s, a) = Q(s, a) - V(s). The Q function, however, measures the the value of choosing a ...


4

The exploration rate, typically parameterized as epsilon / ε, can be changed on every trial. It depends on the complexity of the model and the goals. The simplest thing to do is keep exploration rate high and fixed. That means the model will continue to explore new options, even at the cost of not "exploiting" the best available option. Another option is ...


3

You may be very interested to know that there was a bug in the v2 Lidar tracing, making the agent think there were phantom objects, and sometimes intersecting with its own legs: https://github.com/openai/gym/pull/1789 Finding this bug makes me even more impressed anyone has solved BipedalWalkerHardcore-v2 - it seems the observations from lidar have been ...


3

OpenAI's Gym is a standardised API, useful for reinforcement learning, applied to a range of interesting environments many of which you can then access for free with little effort. It is very simple to use, and IMO worth learning if you want to practice RL using Python to any depth at all. You could use it to ensure you have good understanding of basic ...


3

Welcome to AI.SE @Romain B. I have several undergraduates working on multiagent deep RL problems for their theses, but most of them have been working for 8-9 months. 2 might be a stretch. Good multiagent deep RL problems for a bachelor's thesis might look something like: Pick an older video game, which has been studied using Deep RL, but not in depth. ...


3

Although what @Jaden said may be true by itself, it does not really serve to answer my question as I have seen after conducting numerous experiments, and finally reaching close to Dueling Network performance using a normal Double DQN (DDQN). I made the following changes to my code after closely examining the OpenAI baselines code: Used PongFrameskip-v4 ...


3

Does the opponent's turn affect the calculated rewards? Yes, in general it can. Obvious case, in a two player game where the opponent could win or lose on their turn, but has other options. As far as I know, the reward should only be the result of the agent's action right? In a well-defined MDP, the reward should be a stochastic function of the current ...


3

This is sort of a tricky question to answer, by results you could mean 2 possible things. History, which is a culmination of actions, observations and rewards. Or the value func/policy which is the thing you are trying to improve. With that being said, you for most instances of RL will only be dealing with the last few events in the history, or sampling ...


3

No, there is no file type associated with AI projects in general. Your examples of Photoshop and Excel are specific corporate branded products. These store bespoke data that only works with those products (plus maybe a few converters that can read the files for competitor products). Even more general examples such as .jpg for images or .txt for text ...


2

For example DeepMind [Lab] is not a standard or Open Source friendly I'm not sure where you got that info from... as far as I'm aware, DeepMind Lab is definitely used in various publications (maybe primarily publications from DeepMind, but still). Considering the github repo has the GNU GPL 2 license, it also seems Open Source-friendly to me. Another ...


2

The same argument of a state-value function int Reinforcement Learning: An Introduction Section 13.5 can be applied to state-action values. The main takeaway is that a critic's state value (or action-state value) function is used for bootstrapping. Although the REINFORCE-with-baseline method learns both a policy and a state-value function, we do not ...


2

Deep Reinforcement Learning is a popular answer to representing (state, action) spaces. Instead of having a fixed table of (state, action) pairs, a deep neural network learns a non-linear mapping of the space. Compared to a table, a deep neural network is a more compact representation and is better able to generalize to unseen pairs.


2

OpenAI Retro is an extension to OpenAI Gym. As such, it does not support multiple agents in multi-player environments. Instead, the environment is always presented from the perspective of a single agent that needs to solve a control problem. When there is more than one agent in a Gym problem, everything other than the first agent is considered part of the ...


2

Since the environment has some randomness in it, purely memorizing a trajectory to victory will not work. You will have to memorize every single trajectory for that to work, and there are an infinite number of them. So, you will need to add some sort of bias to your learning model - i.e., what to do when the observations in your pickle file don't match the ...


2

What's exactly the point of time.sleep() in this code? I don't really understand it, you're simply stopping the execution of the program for $0.01$ seconds, how will that affect the simulator in any way ? It's not running in parallel, it does one step of the simulation when you call env.step function and returns the next state and reward. Calling sleep ...


2

The main point in GPT-3 and already in 2 was the observation that performance was steadily increasing with increasing model size (As seen in Figure 1.2 in your linked paper). So it seems that while all progress made in NLP was definitely useful, it also is important to just scale up the model size. This may not seem like surprising point, but it actually ...


2

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are hyper-parameters which basically means they are user specified, whereas ...


2

There is a really small mistake in here that causes the problem: for index, (current_state, action, reward, next_state, done) in enumerate(minibatch): if not done: new_q = reward + DISCOUNT * np.max(future_qs_list) #HERE else: new_q = reward # Update Q value for given ...


1

The video you linked is not using reinforcement learning (RL). It is using genetic algorithms (GA). GA is designed around using multiple agents and picking the best performing to move forward to next generation. With this approach, it is common to want to only view the best performing agents, as the learning mechanism uses the same selection process - the ...


1

I've actually implemented this game before using deep reinforcement learning. You are dealing with a dynamic action space here, where the action space may change at each time step of the game (or more generally the MDP). First, let's discuss the actual action spaces in each one of the two phases of Crib (or Cribbage) and formalize the question. Phase 1: The ...


1

In the DRL nanodegree in Udacity, the instructor says it is possible to combine on- and off-policy learning and suggests the following paper where this has been done: Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (ICLR 2017). Citing the paper: The core idea is to use the first-order Taylor expansion of the critic as a control ...


1

I thought about my input-layer. I had the 500 states one hot encoded. So 499 of every input node would be 0. And 0 is very bad in an neural network. I tried the same code with the "CardPole-v0" and it worked. So think about your input guys


1

I'm not sure why you need a continuing environment, but actually you can make most (if not all) OpenAI Gym environments continuing. When you perform a step, you receive information about the next state, the reward, a termination signal and a dictionary with additional information. Simply ignore the termination signal if you want the episodes to continue ...


1

In general, you can use a simulation to prepare and train a controller for a real world application. A good example of this being done for robotics is in the paper Autonomous helicopter flight via reinforcement learning where a Reinforcement Learning agent was trained on a model of helicopter dynamics before being used in reality. Often, as in this case, ...


1

The key here is think up strategies. If we define this as examining, creating a hypothesis, and testing it as strategizing then yes AI has the ability to strategize. It can examine other users' games, quantifies actions that correlated with victory then test if it gains victory by doing those actions. Strategy by definition is: a plan of action or policy ...


1

So for anyone struggling to understand the OpenAI's Spinning Up educational resource, I'll provide the answer to my question here. Firstly, it's important to understand that the algorithms expect a 2-dimensional input shape, in rudimentary terms a shape of Box(int), which isn't the case with the default Breakout-v0 game environment, which supplies inputs in ...


1

It's not on your end, as a creator of flight simulator, to worry about what action should get the credit for the reward that happened some time after the action was taken. You should return the reward when the actual event happens not when the action that caused it happened. It's the job of the reinforcement learning agent to figure that out. For example if ...


1

This stackoverflow post provides a good answer. It recommends creating the environment as a new package. Note the updated link to the instructions on OpenAI. So that's what I did and it's honestly not that much work and then you can import the environment into your script like this: import gym import gym_foo env = gym.make('foo-v0') This blog post also ...


1

https://github.com/openai/retro Current list of machines is Atari Atari2600 (via Stella) TurboGrafx-16/PC Engine (via Mednafen/Beetle PCE Fast) Game Boy/Game Boy Color (via gambatte) Game Boy Advance (via mGBA) Nintendo Entertainment System (via FCEUmm) Super Nintendo Entertainment System (via Snes9x) GameGear (via Genesis Plus GX) Genesis/Mega Drive (via ...


1

Ok, I found https://blog.openai.com/gym-retro/ but there may be other platforms?


Only top voted, non community-wiki answers of a minimum length are eligible