0
$\begingroup$

I am working in a team to develop a Q-learning based approach for hyperparameter tuning. I have a disagreement with one of my teammates on how they defined this problem. They defined it as follows:

  1. The states are the values of the metric we are optimizing (ex. F1 score: 0, 0.1, 0.2, ..., 1.0).
  2. The actions are to change one of the hyperparameters. So we have as many actions as we have hyperparameters.
  3. The reward is the difference of the metric between the current state and the previous state. We evaluate the model we are optimizing on the current hyperparameter configuration.
  4. Each of the mentioned actions have a sequence of values that we use whenever that action is picked. Ex. The batch size can have values [16,32,64]. When we choose the batch size action, we set it to 16. Next time we choose it, we set it to 32. Next time 64, then we come back to 16.

I have various issues with this formulation. Mainly:

  1. The rewards and Q-values are non-deterministic. Choosing action a in state s, can be either a good or a bad decision, depending on the previous actions we chose, and whatever value this current action is going to return next. From my understanding, this means that the Q-values are useless now. Choosing an action might be good or bad, non-deterministic.
  2. With this reward scheme, the first action will get a high reward no matter what, because there's no previous metric to subtract from it.
  3. I still can't help but feel that this state space is problematic. Choosing action a in state s will also yield an arbitrary next state depending on the evaluated hyperparameter configuration.

Now I have tried getting my point of view across, but no matter how much I argue this person thinks i am wrong. Their use of ChatGPT unfortunately fuels this arrogance as it says yes to every one of their questions, and they treat it as their bible.

Is my argument reasonable? Is there really nothing wrong with this approach? I don't know where else to ask this question

Improve this question
$\endgroup$
2
  • $\begingroup$ We cannot really help you with your colleague's insistence on ChatGPT being correct. They are appealing to "authority" - incorrectly - but will view that you are, too. I recommend you remove that part of your question, as it is a separate issue. There are plenty of resources, including on this site, that warn against using ChatGPT in the way your colleague is - in brief you need to be a better expert than the question you are asking in order to filter out the "hallucinations". Never use ChatGPT to learn or plan a complex subject, the chances of failure are very high (more likely than success). $\endgroup$
    – Neil Slater
    Commented Jan 16 at 10:34
  • $\begingroup$ More practically, on the RL side of your question, it is not clear whether your project is to tune one model, or to create a generic "smart hyperparameter tuner" that could tune multiple models based on observing current F1 score and tweaking hyperparameters? Please update with more details of the goal of the metalearning process (i.e. what are you trying to achieve in this application of RL)? $\endgroup$
    – Neil Slater
    Commented Jan 16 at 10:44

1 Answer 1

Reset to default
1
$\begingroup$

There's a lot being asked here and I don't know that I'm tracking well enough to comment on this formulation. But I will try to clarify some RL theory and answer the title question.

If the transition from one state to the next state is completely non-deterministic, it is not an MDP and a Q-table will likely just give some metric about the underlying random process but not be a reliable predictor.

However, there is likely some non-determinism in most realistic processes (sensor inaccuracy, under/overshoot action, action of unmeasured factors, etc) and Q-learning can still be used. Distributional RL even provides an insight into the non-determinism for greater policy accuracy.

Lastly commenting on your issue 1, just because the current state depends on previous states does not make it non-Markovian or non-deterministic as long as for a certain history and action there is a consistent next transition. In these cases it becomes an n-step MDP. Typical treatments involve a translation to belief states and proceed with Q-learning from there.

Improve this answer
$\endgroup$
5
  • $\begingroup$ define completely non-deterministic $\endgroup$
    – Alberto
    Commented Jan 15 at 14:16
  • $\begingroup$ RL definitely allows non determinism from the transition of the MDP and the reward, indeed they are modeled as random variables $\endgroup$
    – Alberto
    Commented Jan 15 at 14:16
  • $\begingroup$ I’d say that practically completely non-deterministic begins where the tools or evaluation methods used to measure it are no longer sufficient. For a Q-table this can be a function of the granularity of the states for example. $\endgroup$
    – foreverska
    Commented Jan 15 at 18:56
  • $\begingroup$ "where the tools or evaluation methods used to measure it are no longer sufficient" can't get what you are saying... no matter the Rand Var you are using as R/S', maximizing the expectation works just fine $\endgroup$
    – Alberto
    Commented Jan 16 at 12:10
  • $\begingroup$ To go to an extreme, if one wanted to solve woods101 but the representation/q-table had only a single state, no amount of expectation maximization would solve it. It would be practically non-deterministic. This is an extreme example but is true for any problem where granularity or dimensionality of the representation is not sufficient. $\endgroup$
    – foreverska
    Commented Jan 17 at 16:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .