I am looking to have a cooperative multi agent reinforcement learning framework where one agent has a discrete action space and another agent has a continuous action space. Is there a way to do this as most papers I have seen will only handle one or the other.
Seems like what you are looking for is Parametrized RL to train an agent for a Parametrized Markov Decision Process. You can look up both terms to find courses/readings about it.
Anyway, one existing RL framework for it is the Multi-pass Parametrized DQN (MPDQN) which is proposed in this paper, and if google enoug you can even find the author's dissertation on MP-DQN which preliminaries section might help you a lot. In short, the Agent uses Actor-Critic arhitecture in which the Actor will predict the continuous parameters of all actions given the current state, and the Critic will predict the action-value of each action given the states concatenated by the predicted values of the continuous parameters. The final discrete action is chosen by sampling the action-value or using argmax.
In P-DQN, the state concatenated with all the continuous parameters' values are passed to the Critic. However, it is shown that by passing all continuous parameters at once to the Critic, the unused action's parameters will affect the gradient thus affecting the Actor's parameters, which is not expected. Therefore, the parameters of each action are passed separately into the Critic in MP-DQN (unrelated action parameters' are zeroed). With good design, you can implement it so you can pass all the parameters and state in a single pass (multi-pass) as shown in MP-DQN. Other alternative for Parametrized RL is the hybrid-PPO and parametrized DDPG.
Finally, based on my personal experiments, I still have difficulty on controlling the scale of the continuous parameters values prediction so that it stays in a certain predefined range. There are 2 ways to do it, first is using softmax or using gradient inverting (changing the gradients' sign depending on the current predicted parameter values). However, I still eventually produced Actor that output only extreme values (either upper or lower bound) of the continuous parameters.