# Neural network for specific numbers from a range (Q learning)

PROBLEM AT HAND: I have a resource (Bandwidth) of B Hz. I have to distribute the bandwidth B to users as per their requirements. For instance, voice calls would require some amount of bandwidth while video calls would require some amount of bandwidth. Virtual reality applications and Industrial IoT would require more bandwidth than the above two cases. The fraction of bandwidth out of total bandwidth B that I can assign are as follows: [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]. That is to say, anyone among the following fractions can be assigned.

I am trying to design a Reinforcement Learning Agent that would assign bandwidth to different applications based on the amount of bandwidth that a particular application requires.

I am trying to use the following template for my application.

Deep Q Learning

MY QUESTION:

I am not able to understand how can I tell the neural network that it can take only one of the following values:

[0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]

class Agent:
def __init__(self, state_size, action_size):
self.memory = deque(maxlen=2500)
self.learning_rate=0.001
self.epsilon=1
self.max_eps=1
self.min_eps=0.01
self.eps_decay = 0.001/3
self.gamma=0.9
self.state_size= state_size
self.action_size= action_size
self.epsilon_lst=[]
self.model = self.buildmodel()

def buildmodel(self):
model=Sequential()
return model

def add_memory(self, new_state, reward, done, state, action):
self.memory.append((new_state, reward, done, state, action))

def action(self, state):
if np.random.rand() > self.epsilon:
return np.random.randint(0,4)
return np.argmax(self.model.predict(state))

def pred(self, state):
return np.argmax(self.model.predict(state))

def replay(self,batch_size):
minibatch=random.sample(self.memory, batch_size)
for new_state, reward, done, state, action in minibatch:
target= reward
if not done:
target=reward + self.gamma* np.amax(self.model.predict(new_state))
target_f= self.model.predict(state)
target_f[0][action]= target
self.model.fit(state, target_f, epochs=1, verbose=0)

if self.epsilon > self.min_eps:
self.epsilon=(self.max_eps - self.min_eps) * np.exp(-self.eps_decay*episode) + self.min_eps

self.epsilon_lst.append(self.epsilon)

def save(self, name):
self.model.save_weights(name)

I am not able to understand how can I tell the neural network that it can take only one of the following values

You don't have to. The neural network in Deep Q Learning (the DQN) is not configured to output what the actions are. There are a few different possibilities, but in this case the network has been setup to predict the action value of all possible actions as an array of outputs. This is probably the most common setup for a DQN. The DQN doesn't name the actions or associate them with any value from the environment (e.g. in your case the bandwitch allocation fraction). It only knows the actions by their index position in its output.

You are free to associate each action's effect on the bandwidth allocation, 0.0, 0.1, 0.2 etc to arbitary action choices. Provided you do it consistently, it doesn't matter which action index from 0 to 10 gets associated with which item. However, you may as well make a simple intuitive mapping, by dividing the action index by 10.

When interpreting the DQN's output, it might predict for a certain input state e.g. [-0.5, 2.1, 3.7, 1.6, 0.2, 0.5, 0.1, -0.2, -0.5, -0.9, -1.7] - those are the predicted returns from choosing actions. The greedy action from that result would be action index 2, and if you followed the mapping convention I suggested that would be to allocate $$0.2$$ of the bandwidth.

There are other Reinforcement Learning methods that do work directly with numerical actions by predicting either a single number or distribution of numbers that can be used as the action. Q learning does not do that.

def action(self, state):
if np.random.rand() > self.epsilon:
return np.random.randint(0,4)
return np.argmax(self.model.predict(state))

You do not need to tell the network it can only take one of the actions. It is by these lines that a single action is decided upon. Predict returns the Q-values for all of the actions. Argmax gets the action with the highest Q-Value (the highest predict future reward).

As an aside, the randint under epsilon needs to be of the same action_size.