Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
Welcome to AI.SE ! First of all, I'd say that there is a reason to give Learning Rate (LR) and Exploration Rate (ER) the same decay : they play at the same scale (the number of successives batches you'll train your model on). But if I refine the analysis, I would rather say that it's a reason to choose them in the same range, i.e. close to 1, but not specially at the same number.
For LR decay, people often choose it very close to one (which can mean really different things like 0.98 or 0.997), because it plays on a large scale, an you don't want the LR to disappear to brutally.
However, the choice of ER decay can have more variation from model to model. It depends on the initial value of ER (you don't wanna decay fastly ER if you ER is initially low), and also to the "learning speed" of your model : if your model learns efficiently at the beginning, you could want to fastly decrease ER in order to reduce the noise on the action, supposing that you did enough exploration at the beginning (but I think this last opinion is more controversial). You can find an interesting paper here, where the author tries different ER decay and finds out that 0.99 is the best, for CartPole environment.