0
$\begingroup$

I am wanting to train an RL multi-agent model to run in a propietary simulation, which is written in C++. Is there a way to change the simulation itself to create an agent, or must I use a reinforcement learning environment to rebuild my simulation, train the model, and then extract that agent to be used in the simulation? My preferred route would be to change the simulation to enable reinforcement learning, but I have not found examples of that on the web or in literature. Is that possible? Or must the route be that we recreate our simulation in an environment?

$\endgroup$
6
  • $\begingroup$ Please clarify - are you intending to write your own learning agent, and if so do you intend to do this in C++ (with e.g. TensorFlow C++ API for a neural network) or are you hoping to use something pre-built in Python (e.g. one of OpenAI's baseline implementations)? $\endgroup$ Commented Mar 19 at 16:57
  • $\begingroup$ Also important - can the environment and agent interact over reasonably well-defined time steps/decision points, e.g. when training it is possible to move the simulation forward a known amount and use that to generate next state, reward etc? $\endgroup$ Commented Mar 19 at 17:09
  • $\begingroup$ If I can use python, my main language, that would be fantastic. If there is some benefit to using C++, I would use that. I'm not sure of the difficulty of the task, perhaps I would start easy for prototyping, then go hard (Tensorflow C++ API) after that. $\endgroup$ Commented Mar 20 at 0:29
  • $\begingroup$ Regarding the second question, I believe this is possible. Also, I might add, I am planning to do multi-agent RL, with agents that can be of different categories (planes, cruise-liners, air-balloons, etc). $\endgroup$ Commented Mar 20 at 0:31
  • $\begingroup$ According to the question, your environment is written in C++. Does it have an interface that you can script in Python for inputs (agent actions) and outputs (observations of state)? That seems to be what your question is about - how to adapt an RL system around the existing environment. If there is no suitable interface, do you have the C++ source code and able to compile some Python bindings, or only the executable? $\endgroup$ Commented Mar 20 at 4:51

1 Answer 1

0
$\begingroup$

An environment is not a magical concept in Reinforcement Learning (RL). When the textbooks were written on RL the python "gym" concept certainly did not exist much less during the time of some of the earlier research work.

An environment is anything which the agent can sense, interact with and receive feedback from. One could throw an agent into a robotic body an let it learn from the real world in real time. There's no need even for a simulation. It's just common because we can run a simulation at faster than real time and/or spread learning across many agents.

So, for your simulation to be an "environment" it need only be modified to do those three things. Give the agent an input it can sense, accept and enact the agent's actions and evaluate it's performance in the form of a reward signal.

$\endgroup$
4
  • $\begingroup$ I see. I had come to understand it would take an interface that would output a state vector. However, I would like to use a multi-agent RL technique, with varying types of units. One step would be full information, knowing the whole state of the environment to decide initial placements. And the future steps would be partial. It would have interesting things like deploying an energy capture device, with the reward delayed. But from what you say it seems like you would just modify or replace the current algorithms used to generate each units behavior. $\endgroup$ Commented Mar 20 at 0:44
  • $\begingroup$ So yes, the environment must present the agent with a state vector. That is the "sensing" part. One might have an agent which is the general, pilot and sailor all at once, sensing all of the information simultaneously. But it is also possible for each to be their own agent, with their own collection of things they can sense. Which is better is something to test. $\endgroup$
    – foreverska
    Commented Mar 20 at 2:14
  • $\begingroup$ What about a battleship, a plane, and a missile launcher, MARL can handle all of that? They'll all have partial information, communicated to each other in timesteps. Then the only question I have is how do I do this. Is there a guide, a paper? $\endgroup$ Commented Mar 20 at 3:23
  • $\begingroup$ I don’t have a lot of MARL experience. I found a couple interesting papers starting here, hope it helps. arxiv.org/abs/1912.03558 $\endgroup$
    – foreverska
    Commented Mar 20 at 12:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .