The environment of the Clean Up game is simple: in a 25*18 grid world, there's dirt spawning on the left side and apples spawning on the other. Agents get a +1 reward for eating an apple (by stepping onto it). Agents clean the dirt also by stepping on it (no reward). Agent can go up, down, left, right. The game goes on for 1000 steps. Apple's spawn probability depends on the amount of dirt (less dirt, higher the probability). Currently, the observation for each agent has the manhatten distance to their closest apple and dirt.

I have tried multiple ways of training this, including changing the observation space of the agents. But it seems the result does not outperform random agents by any significant amount.

The network is simple, it tries to take in all the observations for all the agents and give the reward predictions for each action for all agents:

def simple_model():
    input = Input(shape=(num_agents_cleanup, 8))
    flat_state = Flatten()(input)

    layer1 = Dense(512, activation = 'linear')(flat_state)

    layer2 = Dense(256, activation = 'linear')(layer1)
    layer3 = Dense(64, activation="relu")(layer2)
    actions = Dense(4*num_agents_cleanup, activation="linear")(layer3)
    action = Reshape((num_agents_cleanup, 4))(actions)
    return Model(inputs=input, outputs=action)

I haven't had much experience and trying to learn MARL so there could be some fundamental mistakes here. Anyways the training mainly look like this:

batch_size = 32
for i_episode in range(num_episodes):
    states, _ = env_qd.reset()
    eps *= eps_decay_factor
    terminate = False
    num_agents = len(states)
    mem = []  # memorize the steps
    while not terminate:
        # env_qd.render()
        actions = {}
        comb_state = []
        for i in range(num_agents_cleanup):
            comb_state.append(states[str(i)])  # combine the states for all agents
        comb_state = np.array(comb_state)
        a = model_simple.predict(comb_state.reshape(1, num_agents_cleanup, 8), verbose=0)[0]
        for i in range(num_agents):
            if np.random.random() < eps:
                actions[str(i)] = np.random.randint(0, env_qd.action_space.n)
                actions[str(i)] = np.argmax(a[i])
        new_states, rewards, done, _, _ = env_qd.step(actions)
        new_comb_state = []
        for i in range(num_agents_cleanup):
            new_comb_state.append(new_states[str(i)])  # combined new state
        new_comb_state = np.array(new_comb_state)
        new_pred = model_simple.predict(new_comb_state.reshape(1, num_agents_cleanup, 8), verbose=0)[0]
        target_vector = a

        for i in range(num_agents):
            target = rewards[str(i)] + discount_factor * np.max(new_pred[i])
            target_vector[i][actions[str(i)]] = target
        mem.append((comb_state, target_vector))
        states = new_states
        terminate = done["__all__"]
    for i in range(35):
        minibatch = random.sample(mem, batch_size)  # trying to do experience replay
        state_batch = []
        target_batch = []
        for i in range(len(minibatch)):
        np.array(state_batch).reshape(batch_size, num_agents_cleanup, 8),
        np.array(target_batch).reshape(batch_size, num_agents_cleanup, 4),
        epochs=1, verbose=0)

The training would start to learn something at first (it seems), but then slowing "converge" to a very low reward.


discount_factor = 0.99
eps = 0.3
eps_decay_factor = 0.99

Is there any glaring mistake that I made in the training process? Is there a good way to define the agents' observations?

Thank you!



You must log in to answer this question.