0
$\begingroup$

The initial environment state is 0.25. Each time step the agent performs a discrete action of 0 or 1. If action is 1, then the new state will be state + 0.1. If action is 0, the new state will be state - random() * 0.2. The reward is state - 0.5, however if state > 0.98 (or state < 0) the agent dies (with no reward).

First question: How do I teach the agent not to be too greedy? How to verify that the agent learned?

Main question: How to reduce the number of trials (i.e. the number of episodes) before the agent learns?

I would also appreciate any relevant references.

Here is the environment and here is what I tried.

It works, however:

  1. It took 1000 episodes of max 2000 timesteps, which is unacceptable for me (I wish to drastically reduce the number of episodes and timesteps).

  2. The behavior is far from optimal. Ideally, the agent should choose action 0 only if the state is larger than 0.88 (or something below that and within a small interval such as 0.01). [Edit] However, the threshold is 0.75, that forces the agent to choose 0 even if it could safely choose 1, e.g. following 0.8 -> 0.76 -> 0.75 -> 0.74 trajectory before choosing 1 again.

$\endgroup$
2
  • $\begingroup$ Some ideas to reduce the amount of direct interaction with an environment: behavior cloning, off-policy algorithms. By observing, rather than interacting, these algorithms should partially alleviate the amount of trial and error. $\endgroup$
    – penkovsky
    Aug 10 at 12:30
  • $\begingroup$ Which is great if it is extremely costly to simulate an environment. $\endgroup$
    – penkovsky
    Aug 10 at 12:37
1
+50
$\begingroup$

TLDR: Simplify your agent.

Context:

As you've noticed, it's not a hard game and it does not require a complex policy.

You'd need a single neuron to solve it perfectly: 1 if (state < 0.78) else 0

Problem:

However, the default agent comes with a big neural network (I believe it's a 2 layered 64 fully connected neural network). So you have thousands of parameters trying to solve a simple problem. And that's why it takes thousands of time-steps.

Solution:

So first thing I'd recommend is to use a simpler policy model.

If it doesn't completely solve the problem, then simplify your model by diving deeper on your agent and disabling the functions you think aren't helpful. https://tensorforce.readthedocs.io/en/latest/modules/policies.html

Lastly, if you really need to nail it, you can use some meta-learning for tuning the hyper-parameters.


Edit: adding Meta-Learning

Meta-Learning is a broad concept of using machine learning for setting up your machine learning architecture and/or hyperparameters. It's an automation of your trial and error process, exploring and exploiting different configurations in a search for a good model.

A related concept is auto-ML.

Keep in mind that:

  1. It's a very extensive computational process, since you need to train and evaluate thousand or millions of models.
  2. You'll need to explicitly define your definition of a good model: Will you reward it to be simple? Are you looking for fast training speed? Does in need to be light for deploying? Or you want a huge, complex, but accurate model?

In your case, if the training time is too big, you could limit the training time, so you'll find the best model that can be trained in a fixed short time window. And once again, keep in mind it will take way more time to find a good model this way, than just selecting a not ideal architecture and train it extensively.

Here is a Siraj's video talking about the general concept.

$\endgroup$
8
  • $\begingroup$ This is a valid point, indeed. Can you please explain more about the meta-learning, as it seems to be directly relevant to the main question. Thank you $\endgroup$
    – penkovsky
    Aug 13 at 9:54
  • $\begingroup$ I actually tried to reduce the number of layers to a single hidden layer and reducing the number of neurons. Doesn't help much. $\endgroup$
    – penkovsky
    Aug 13 at 9:58
  • $\begingroup$ For instance, after training a single layer of 16 neurons (with different number of timesteps and episodes), during evaluation, the agent dies after about 10 timesteps. $\endgroup$
    – penkovsky
    Aug 13 at 10:02
  • $\begingroup$ I've updated the orignal post with meta-learning brief concepts. $\endgroup$ Aug 13 at 20:36
  • 1
    $\begingroup$ As an appreciation of your effort, I decided to award the reputation bounty anyway. Thank you. $\endgroup$
    – penkovsky
    Aug 16 at 11:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.