0
$\begingroup$
  • Are there any examples for RL to learn analytical functions (for example, $y=x^2$)?
  • What are the considerations when constructing the environment?
  • Are there any literature that analyzes the difficulty/validity of the problem?
  • Which algorithm is best for this type of problem? Why?
class Env(gym.Env):
    def __init__(s, config = {}):
        s.observation_space = spaces.Box(0, 1, [1, ])
        s.action_space = spaces.Box(0, 1, [1, ])
    
    def reset(s):
        s.state = np.random.rand(1)
        return s.state
    
    def step(s, action):
        reward = - abs(s.state ** 2 - action)[0]
        return s.state, reward, True, {}
  • trained with PPO for 100 iterations, result NOT good:
+---------------------+------------+-------+--------+------------------+--------+------------+----------------------+----------------------+--------------------+
| Trial name          | status     | loc   |   iter |   total time (s) |     ts |     reward |   episode_reward_max |   episode_reward_min |   episode_len_mean |
|---------------------+------------+-------+--------+------------------+--------+------------+----------------------+----------------------+--------------------|
| PPO_Env_e2752_00000 | TERMINATED |       |    100 |          815.105 | 400000 | -0.0105903 |         -1.54313e-08 |            -0.262791 |                  1 |
+---------------------+------------+-------+--------+------------------+--------+------------+----------------------+----------------------+--------------------+

enter image description here

$\endgroup$
8
  • 2
    $\begingroup$ What is your purpose for doing this? RL is a bad choice for learning functions in the abstract, you may as well collect the same number of observations for supervised learning if the next state is not under the agent's control. $\endgroup$ Aug 3 '21 at 8:03
  • $\begingroup$ my purpose is to understand RL better. The direct motivation is: suppose I provide observation $x$ to the agent, from educated interpretation, it would be helpful to provide the agent with $x^2$, I wonder whether I need to provide $x^2$ or can the agent figure out itself $\endgroup$
    – Rick
    Aug 3 '21 at 10:26
  • $\begingroup$ for example, the direct observation is an array arr with shape [n, ], suppose I design the environment such that the best action is argmax(arr**2), can the agent figure out itself or do I calculate arr**2 for the agent as observation? $\endgroup$
    – Rick
    Aug 3 '21 at 10:29
  • $\begingroup$ To understand RL better, you should use one of the standard RL toy environments. Your environment design is missing key things that RL is made to solve, such as time steps and state progression. Try CartPole, Taxi, FrozenLake or similar simple environments in OpenAI gym $\endgroup$ Aug 3 '21 at 10:39
  • $\begingroup$ To partially answer your title, then yes RL can be used to learn unknown analytical functions, provided an agent is allowed to operate the function as a "black box" and observe results. However, this is because RL is a very general learning algorithm that can be made to fit many scenarios. It is not the best choice, and not even a very good choice for the problem as you have framed it. You will not learn much about the normal use of RL by solving this problem with RL methods. $\endgroup$ Aug 3 '21 at 10:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.