How to reduce the number of episodes before the agent learns in this game?

Question

The initial environment state is 0.25. Each time step the agent performs a discrete action of 0 or 1. If action is 1, then the new state will be state + 0.1. If action is 0, the new state will be state - random() * 0.2. The reward is state - 0.5, however if state > 0.98 (or state < 0) the agent dies (with no reward).

First question: How do I teach the agent not to be too greedy? How to verify that the agent learned?

Main question: How to reduce the number of trials (i.e. the number of episodes) before the agent learns?

I would also appreciate any relevant references.

Here is the environment and here is what I tried.

It works, however:

1. It took 1000 episodes of max 2000 timesteps, which is unacceptable for me (I wish to drastically reduce the number of episodes and timesteps).

2. The behavior is far from optimal. Ideally, the agent should choose action 0 only if the state is larger than 0.88 (or something below that and within a small interval such as 0.01). [Edit] However, the threshold is 0.75, that forces the agent to choose 0 even if it could safely choose 1, e.g. following 0.8 -> 0.76 -> 0.75 -> 0.74 trajectory before choosing 1 again.

Answer 1

TLDR: Simplify your agent.

Context:

As you've noticed, it's not a hard game and it does not require a complex policy.

You'd need a single neuron to solve it perfectly: 1 if (state < 0.78) else 0

Problem:

However, the default agent comes with a big neural network (I believe it's a 2 layered 64 fully connected neural network). So you have thousands of parameters trying to solve a simple problem. And that's why it takes thousands of time-steps.

Solution:

So first thing I'd recommend is to use a simpler policy model.

If it doesn't completely solve the problem, then simplify your model by diving deeper on your agent and disabling the functions you think aren't helpful. https://tensorforce.readthedocs.io/en/latest/modules/policies.html

Lastly, if you really need to nail it, you can use some meta-learning for tuning the hyper-parameters.

---

Edit: adding Meta-Learning

Meta-Learning is a broad concept of using machine learning for setting up your machine learning architecture and/or hyperparameters. It's an automation of your trial and error process, exploring and exploiting different configurations in a search for a good model.

A related concept is auto-ML.

Keep in mind that:

1. It's a very extensive computational process, since you need to train and evaluate thousand or millions of models.
2. You'll need to explicitly define your definition of a good model: Will you reward it to be simple? Are you looking for fast training speed? Does in need to be light for deploying? Or you want a huge, complex, but accurate model?

In your case, if the training time is too big, you could limit the training time, so you'll find the best model that can be trained in a fixed short time window. And once again, keep in mind it will take way more time to find a good model this way, than just selecting a not ideal architecture and train it extensively.

Here is a Siraj's video talking about the general concept.