Hot answers tagged

4

The exploration rate, typically parameterized as epsilon / ε, can be changed on every trial. It depends on the complexity of the model and the goals. The simplest thing to do is keep exploration rate high and fixed. That means the model will continue to explore new options, even at the cost of not "exploiting" the best available option. Another option is ...


4

OpenAI's Gym is a standardised API, useful for reinforcement learning, applied to a range of interesting environments many of which you can then access for free with little effort. It is very simple to use, and IMO worth learning if you want to practice RL using Python to any depth at all. You could use it to ensure you have good understanding of basic ...


4

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are hyper-parameters which basically means they are user specified, whereas ...


3

No, there is no file type associated with AI projects in general. Your examples of Photoshop and Excel are specific corporate branded products. These store bespoke data that only works with those products (plus maybe a few converters that can read the files for competitor products). Even more general examples such as .jpg for images or .txt for text ...


3

Does the opponent's turn affect the calculated rewards? Yes, in general it can. Obvious case, in a two player game where the opponent could win or lose on their turn, but has other options. As far as I know, the reward should only be the result of the agent's action right? In a well-defined MDP, the reward should be a stochastic function of the current ...


3

You may be very interested to know that there was a bug in the v2 Lidar tracing, making the agent think there were phantom objects, and sometimes intersecting with its own legs: https://github.com/openai/gym/pull/1789 Finding this bug makes me even more impressed anyone has solved BipedalWalkerHardcore-v2 - it seems the observations from lidar have been ...


3

Although what @Jaden said may be true by itself, it does not really serve to answer my question as I have seen after conducting numerous experiments, and finally reaching close to Dueling Network performance using a normal Double DQN (DDQN). I made the following changes to my code after closely examining the OpenAI baselines code: Used PongFrameskip-v4 ...


3

Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value V(s) function is estimated separately from the state-action value Q(s, a). A new quantity, the advantage of an action, can then be defined as A(s, a) = Q(s, a) - V(s). The Q function, however, measures the the value of choosing a ...


3

This is sort of a tricky question to answer, by results you could mean 2 possible things. History, which is a culmination of actions, observations and rewards. Or the value func/policy which is the thing you are trying to improve. With that being said, you for most instances of RL will only be dealing with the last few events in the history, or sampling ...


3

There is a really small mistake in here that causes the problem: for index, (current_state, action, reward, next_state, done) in enumerate(minibatch): if not done: new_q = reward + DISCOUNT * np.max(future_qs_list) #HERE else: new_q = reward # Update Q value for given ...


2

In general, the approach of adding or altering the reward structure of a RL environment, when you are still trying to create an agent that solves the original problem, is called Reward Shaping. Reward Shaping can be tricky to get right. In your case I think it may be counter-productive. The main issue is that the agent does not know the current time step as ...


2

For example DeepMind [Lab] is not a standard or Open Source friendly I'm not sure where you got that info from... as far as I'm aware, DeepMind Lab is definitely used in various publications (maybe primarily publications from DeepMind, but still). Considering the github repo has the GNU GPL 2 license, it also seems Open Source-friendly to me. Another ...


2

The same argument of a state-value function int Reinforcement Learning: An Introduction Section 13.5 can be applied to state-action values. The main takeaway is that a critic's state value (or action-state value) function is used for bootstrapping. Although the REINFORCE-with-baseline method learns both a policy and a state-value function, we do not ...


2

OpenAI Retro is an extension to OpenAI Gym. As such, it does not support multiple agents in multi-player environments. Instead, the environment is always presented from the perspective of a single agent that needs to solve a control problem. When there is more than one agent in a Gym problem, everything other than the first agent is considered part of the ...


2

Since the environment has some randomness in it, purely memorizing a trajectory to victory will not work. You will have to memorize every single trajectory for that to work, and there are an infinite number of them. So, you will need to add some sort of bias to your learning model - i.e., what to do when the observations in your pickle file don't match the ...


2

What's exactly the point of time.sleep() in this code? I don't really understand it, you're simply stopping the execution of the program for $0.01$ seconds, how will that affect the simulator in any way ? It's not running in parallel, it does one step of the simulation when you call env.step function and returns the next state and reward. Calling sleep ...


2

The main point in GPT-3 and already in 2 was the observation that performance was steadily increasing with increasing model size (As seen in Figure 1.2 in your linked paper). So it seems that while all progress made in NLP was definitely useful, it also is important to just scale up the model size. This may not seem like surprising point, but it actually ...


1

I don't recommend changing the rules of the environment. What you could do: Perform a method called bucketing i.e. take a value from a continuous state space see which discrete bucket it should go into and then let your agent use the bucket number as the observation. e.g. Say I do have a continuous state space with one variable in range $[-\infty,\infty]$ ...


1

If I understand your problem correctly, you can test on just about any environment, and just omit parts of the observations to ensure your RNN is learning. For example, you can test on cartpole, ignoring the velocity and angular velocity states. This way the MDP isn't actually Markovian and you'll need the RNN to learn.


1

The video you linked is not using reinforcement learning (RL). It is using genetic algorithms (GA). GA is designed around using multiple agents and picking the best performing to move forward to next generation. With this approach, it is common to want to only view the best performing agents, as the learning mechanism uses the same selection process - the ...


1

I've actually implemented this game before using deep reinforcement learning. You are dealing with a dynamic action space here, where the action space may change at each time step of the game (or more generally the MDP). First, let's discuss the actual action spaces in each one of the two phases of Crib (or Cribbage) and formalize the question. Phase 1: The ...


1

In the DRL nanodegree in Udacity, the instructor says it is possible to combine on- and off-policy learning and suggests the following paper where this has been done: Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (ICLR 2017). Citing the paper: The core idea is to use the first-order Taylor expansion of the critic as a control ...


1

What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own reward. I represent my RL agents' actions as dict, containing the RL agent ID as key and its action as value. The different agents may either use the same or a ...


1

I thought about my input-layer. I had the 500 states one hot encoded. So 499 of every input node would be 0. And 0 is very bad in an neural network. I tried the same code with the "CardPole-v0" and it worked. So think about your input guys


1

I'm not sure why you need a continuing environment, but actually you can make most (if not all) OpenAI Gym environments continuing. When you perform a step, you receive information about the next state, the reward, a termination signal and a dictionary with additional information. Simply ignore the termination signal if you want the episodes to continue ...


1

In general, you can use a simulation to prepare and train a controller for a real world application. A good example of this being done for robotics is in the paper Autonomous helicopter flight via reinforcement learning where a Reinforcement Learning agent was trained on a model of helicopter dynamics before being used in reality. Often, as in this case, ...


1

The key here is think up strategies. If we define this as examining, creating a hypothesis, and testing it as strategizing then yes AI has the ability to strategize. It can examine other users' games, quantifies actions that correlated with victory then test if it gains victory by doing those actions. Strategy by definition is: a plan of action or policy ...


1

So for anyone struggling to understand the OpenAI's Spinning Up educational resource, I'll provide the answer to my question here. Firstly, it's important to understand that the algorithms expect a 2-dimensional input shape, in rudimentary terms a shape of Box(int), which isn't the case with the default Breakout-v0 game environment, which supplies inputs in ...


1

It's not on your end, as a creator of flight simulator, to worry about what action should get the credit for the reward that happened some time after the action was taken. You should return the reward when the actual event happens not when the action that caused it happened. It's the job of the reinforcement learning agent to figure that out. For example if ...


1

This stackoverflow post provides a good answer. It recommends creating the environment as a new package. Note the updated link to the instructions on OpenAI. So that's what I did and it's honestly not that much work and then you can import the environment into your script like this: import gym import gym_foo env = gym.make('foo-v0') This blog post also ...


Only top voted, non community-wiki answers of a minimum length are eligible