1
$\begingroup$

I'm trying to build a DQN Agent to take a set of 10 best actions simultaneously (integer values from 1 to 100) as outputs per episode. The input is a float. The goal is to find the optimal combination of (10) actions per episode.

Currently, the set up is having a single NN output 10 actions w/ the highest q-valules for each episode. But in the Memory Replay process, each individual Set (of 10 fixed actions obtained from the exploration phase) is being treated as a single action. Because the target network also takes the output of the list of 10-action from the main NN. Hence I can see the agent repeatedly trying certain Set (with a fixed 10 actions) in the replay/retrain part, whereas our goal is to find the optimal combination of 10 actions per episode, Not the optimal Set of fixed combinations. So in essence, I would like the agent to pick out and mix up the actions from the Sets with higher Q-values (known from the exploration phase) to form new optimal "Sets" in the Replay process.

I was thinking maybe instead of using a single NN with 10 outputs I could do 10 NNs with single outputs for each episode so that each action is treated separately. And I suppose I will have 10 q-networks and target networks as well, then I could combine the results by the end of each episodes. But, I am not sure if that is necessarily the best way to fix the problem of having repetitive sets of fixed action in the replay process.

Alternatively, I think the problem could be treated as a multi-armed bandit problem, except each arm here has "sub-arms" too so to speak, but that could require some changes to the custom environment I am working with and I don't want to touch that unless necessary.

Maybe there is a clever manipulation within the retrain process given my current setup that I am not seeing. Here is a snippet of the code for some more clarity.

class DQNAgent():

    def __init__(self,optimizer):
        # Initialize atributes
        self._state_size = 1
        self._action_size = 76
        self._optimizer = optimizer
        
        self.experience_replay = deque(maxlen=2000)
        
        # # Initialize discount and exploration rate
        # self.gamma = 0.6
        # self.epsilon = 0.5
        self.gamma = 0.95
        self.epsilon = 1.0
        self.epsilon_min = 0.01
        self.epsilon_decay = 0.95
        self.learning_rate = 0.01
        
        # Build networks
        self.q_network = self._build_compile_model()
        self.target_network = self._build_compile_model()
        
    
    def store(self, state, action, reward, next_state, terminated):
        self.experience_replay.append((state, action, reward, next_state, terminated))
        
    def _build_compile_model(self):
        
        model = Sequential()
        model.add(InputLayer(input_shape=(self._state_size,)))
        model.add(Dense(100, activation='relu'))
        model.add(Dense(100, activation='relu'))
        model.add(Dense(self._action_size, activation='linear'))
        model.compile(loss='mse', optimizer=self._optimizer)
        return model
    
    def alighn_target_model(self):
        self.target_network.set_weights(self.q_network.get_weights())

    def retrain(self, batch_size):
        if len(self.expirience_replay) < batch_size:
            return
        minibatch = random.sample(self.expirience_replay, batch_size)
    
        for state, action, reward, next_state, terminated in minibatch:
            
            target = self.q_network.predict(np.reshape(np.array(state), (-1,1)))
            print('target size :', np.shape(target))
            
            
            if terminated:
                target[0][action] = reward
            else:
                t = self.target_network.predict(np.reshape(np.array(next_state), (-1,1)))
                target[0][action] = reward + self.gamma * np.amax(t)
    
            self.q_network.fit(np.reshape(np.array(state), (-1,1)), target, epochs=1, verbose=0)
        
    
    def act(self,state):
        self.epsilon *= self.epsilon_decay
        self.epsilon = max(self.epsilon_min, self.epsilon)
        action_space = [ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
       35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
       52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68,
       69, 70, 71, 72, 73, 74, 75, 76] #all 76 available nodes
        if np.random.rand() <= self.epsilon:
            return np.array(random.sample(action_space,10))-1 #-1 to match control's index

        q_values = self.q_network.predict(np.reshape(np.array(state), (-1,1)))
        print("q_vals shape",np.shape(q_values))
        print('q_vals type',type(q_values))
  
        top_actions_idx = q_values[0].argsort()[-10:][::-1]
$\endgroup$
0

1 Answer 1

0
$\begingroup$

But, I am not sure if that is necessarily the best way to fix the problem of having repetitive sets of fixed action in the replay process.

The best way to handle this is using a Multi Discrete Action Space. You don't need a neural network for every action on its own.

$\endgroup$
2
  • $\begingroup$ Hi @tnfru, thanks for your suggestions. I'm not really familiar with this concept. I am under the impression that for each action taken, other actions in the action space would be impacted in someway. Could I simultaneously sample actions from several different action space for every single episode? And if I were to use Multi Discrete Action Space, would it be necessary to adopt methods like actor-critic instead? $\endgroup$
    – Rkz
    Aug 29, 2021 at 17:22
  • $\begingroup$ Yes @Rkz , DQN will work on discrete actions and Multi Discrete allows you to select from a set of values per action dimension and thus your action space is discrete. You don't need to switch methods per se, but it is very unual to use DQN for Multi Discrete so I would recommend the switch yes. $\endgroup$
    – tnfru
    Aug 30, 2021 at 10:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .