In the Pursuit algorithm (to balance exploration and exploitation), the greedy action has a probability say $p_1$ (updated every episode) of being selected, while the rest have a probability $p_2$ (updated every episode) of being selected.

Could you please show me an example code (Python) on how to enforce such conditional probabilistic picking?

  • $\begingroup$ numpy.random.choice lets you do weighted random choice, study the docs for more info. $\endgroup$ – Brale Jul 19 '19 at 9:04
  • $\begingroup$ numpy.random.choice expects a probability distribution such that if the probability of picking the elements are added individually, it would sum to 1. But in this one, a particular value in the list has a probability say p =0.7 while all the rest have p = 0.3. If you try numpy.random.choice, it returns an error since the distribution is greater than 1. $\endgroup$ – EArwa Jul 19 '19 at 10:14
  • 1
    $\begingroup$ action probabilities need to sum up to 1. You cannot have probability higher than 1. You misunderstood the part about picking other actions. If greedy action is picked with probability of 0.7 then other actions together have probability 0.3 of being picked, meaning each other action has probability of $\frac{0.3}{n-1}$ of being picked. $\endgroup$ – Brale Jul 19 '19 at 10:20
  • $\begingroup$ also in pursuit algorithm its not guaranteed that other non greedy actions have same probability of being picked, probabilities are updated each on their own, but all action probabilities always need to sum up to 1 $\endgroup$ – Brale Jul 19 '19 at 10:21
  • $\begingroup$ Okay, so assuming I did say N pickings (for N sufficiently large), and say the greedy action is picked n1 times while the rest picked n2 times; will I get n1/N = 0.7 and n2/N = 0.3? $\endgroup$ – EArwa Jul 19 '19 at 12:53

If I am understanding your question properly, you could use something like the following:

import numpy as np 
p1 = 0.1
if np.random.rand() < p1: 
    action = 'greedy'
    action = np.random.choice(['other_policy1', 'other_policy2', 'other_policy3']) 

return action 

  • $\begingroup$ Okay, what I have is like a list of actions say x = [1, 5, 6, 7, 8, 9, 10], and let us say the greatest element in the list has a probability p = 0.7 of being picked and all the rest have a probability p = 0.3 of being picked. $\endgroup$ – EArwa Jul 19 '19 at 10:12
  • 1
    $\begingroup$ what do you mean by greatest? It could be that you have a list of actions, and each action has a probability of being picked ``` actions = [1,3,4] np.random.choice(actions, p=[0.7, 0.15, 0.15]) ``` If by greatest you mean the greatest value (in your example, 10), then you could use ``` greatest_element = np.max(x) if np.random.rand() < 0.7: action = greatest_element else: action = np.random.choice(list(set(a) - {greatest_element})) ``` $\endgroup$ – yahiaelgamal Jul 19 '19 at 16:05
  • $\begingroup$ Like in the list, the greatest element is 10. $\endgroup$ – EArwa Jul 19 '19 at 16:07
  • 1
    $\begingroup$ docs.scipy.org/doc/numpy/reference/generated/… generates a random number between 0-1 $\endgroup$ – yahiaelgamal Jul 19 '19 at 16:14
  • 1
    $\begingroup$ I highly recommended reviewing basic probability theory in order to avoid being deceived by randomness. $\endgroup$ – yahiaelgamal Jul 21 '19 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.