0
$\begingroup$

In reinforcement learning problems like cartpole, usually the environment is a single system that takes an input and gives an output. For example, in cartpole, given positions, velocities as input we can tell if pole is going to fall. Hence, we can craft observation space from positions and velocities.

But if environment is made of sequential blocks of systems {A, B, C}, where input is given to A that passes to B then to C then to output, then how do we craft observations?

Options:

  1. We use a separate block D (linear or non-linear) to create numerical observations for input which is then fed to A. Problem with this: Observation crafted out of D is not correlated to output at C. For example, in cartpole, I would say that I would pass the velocities and positions thru a block D to get amount of fuel left in cart. And I would feed the amount of fuel left as observation to actual system which determines if the pole falls.
  2. We use block A to create numerical observations for input. Better than option 1 but observations crafted out of A are not reflective of the entire process (A -> B -> C).

So, how do we craft observations for such an RL problem?

Edit: An example scenario. I have a system A that takes a song and gives N similar songs as output. I have a system B that takes a song and determines if the song is liked by more than 10 million people. Then I have a system C that takes a song liked by more than 10 million people and says if the song is of genre 'pop' or 'edm'.

I have 10 'edm' songs (liked by more than 10 million people) to start with, and I want to apply Reinforcement Learning to expand the 10 'edm' 'liked by more than 10 million people' songs to more such songs. By optimally, I mean we could always brute force for each song but the system A is non-deterministic. It can give N similar songs (N ranges from 0 to 100). In X amount of time, I want to maximize creations of such songs. Say, song X and song Y are the initial inputs. After we pass them thru A->B->C we could get 100 such similar ('edm' & 'liked by more than 10 million people') songs for X but only 2 for Y. Brute-force would waste time a lot in places that would not yield many such songs.

Now, an RL agent can decide which song to pick to get maximum number of similar such songs in X amount of time. But the environment is A -> B (is it liked by 10 million people, output is 1 or 0) -> C (is it 'edm' or 'pop', output is 1 or 0). Songs are represented as signals (array of continuous values between -inf to +inf) by passing them thru a system D. The system D takes the song name, downloads it and gets the signals (array representation of the songs). So, these numerical representations can be used instead of song names.

How do I craft observations for this combined environment?

As far as I know observation space should be such that given 2 observations we can distinguish which one gives better results, like in cartpole, 2 sets of observations (positions1, velocities1) and (positions2, velocities2) the system can identify which observation is relatively better or worse. But in our system (A -> B -> C) how do we go about crafting an observation that correlated with output?

Improve this question
$\endgroup$
4
  • $\begingroup$ you could model this as a graph and use a graph neural network to process the state. each block would be a node, and edges would represent relationships between the nodes. this has been covered somewhat in the literature (relational RL) $\endgroup$
    – David
    Commented Apr 19, 2022 at 13:22
  • $\begingroup$ I am not sure I understand your problem. Could you give a concrete example of a system which consists of these separate blocks that you are concerned about, and explain why you think there is some issue (are you not able to receive or measure state-related information from some of the blocks, if not then why not?) $\endgroup$
    – Neil Slater
    Commented Apr 19, 2022 at 14:08
  • $\begingroup$ Thanks for the update, it helps understand what you mean by "block". I don't have any idea for an answer though, sorry. I do think the "block" thing is not the real problem here - your state can just be (a summary of) the songs selected so far, because everything else is either random or deterministic functions that will give you measurable results once you have chosen your songs. So RL would be learning the output of (likely a very complex) return-prediction function based on the song characteristics. I think the real problem is trying to learn this function at all through trial and error. $\endgroup$
    – Neil Slater
    Commented Apr 19, 2022 at 15:40
  • $\begingroup$ Wont RL learn to map the system D (the machine that takes input song name and produces signal [array/numerical representation]) output to the final output (at C). So, if I train RL agent on 10 songs. And I try to predict on 10 unseen songs, won't the agent perform poorly? Agent learnt actions, values based on numerical representation of system D on 10 train songs. 10 test unseen songs may have num representations not similar/closer to training ones. Agent will perform poorly. It hasn't seen those num representations. Do we then train on large number of songs with varied representations? $\endgroup$
    – Prasanjit Rath
    Commented Apr 19, 2022 at 15:56

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .