2
$\begingroup$

Can an RL algorithm trained in one environment be successful in a different one?

For example, if I train a model to go through one labyrinth, could this model also go through a different but similar labyrinth or would it need a new training process?

By similar, I mean like these two:

enter image description here

enter image description here

But with this one being not similar:

enter image description here

$\endgroup$
  • 1
    $\begingroup$ What is your measure of similarity? I think you should add some details about what is similar and dissimilar between the two labyrinths. Also I think it would matter on how you trained the RL algo, penalty due to some action in 1 labyrinth might not be the same in another labyrinth, so during training if you took the penalty of 1st labyrinth into account it might lead to differences. $\endgroup$ – DuttaA Sep 13 at 10:05
  • $\begingroup$ I used labyrinth only as as an example to more general aspect, but yeah, I edited the question to explain a bit more. $\endgroup$ – Makintosz Sep 13 at 10:19
0
$\begingroup$

Can an RL algorithm trained in one environment be succesfull on a different one?

Strictly the answer here is "no". You train an agent to solve a single environment. If a second environment is similar enough, you can probably re-train the agent on the new environment with little or no changes to the agent, and to get results just as good in the second agent.

In some cases, you could even start with the trained agent and "fine tune" it to the new environment. That probably would not work well with the labyrinth example though, especially if the start and end points are moved.

Example: If I train a model to go through one labyrinth, could this model also go through different but similar labyrinth or would it need a new training process?

This could be subtly different. It is down to how you define the environment. It is possible to define an environment not as "this maze" but for example "all possible mazes in a 30x20 grid pattern". To do so, you need to make the maze configuration part of the environment's state.

Expanding from a singular environment example to all similar environments has a cost. In the labyrinth example, this is significant. A single 30x20 maze has 600 states, which is a trivial problem in RL. All possible mazes of the same size has closer to $2^{1200}$ states, which is much larger and requires different kinds of approaches, different RL methods.

You would expect to train on many example mazes (typically a randomly generated new one for each episode) and use some kind of generalisation - e.g. a neural network, maybe a CNN due to the grid design - when handling the value function or policy function. Training time for a single maze will be under a second, and the policy will be perfect. Training time for all possible mazes measured in hours or days and the policy will still get things wrong from time-to-time.

$\endgroup$
  • $\begingroup$ Labyrinth was just the easiest example I could think of, but I see the point now :) $\endgroup$ – Makintosz Sep 14 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.