This is the problem: I have 66 slot-machines and for each of them I have 7 possible actions/arms to choose from. At each trial, I have to choose one of 7 actions for each and every one of the 66 slots. The reward depends on the combination of these actions but the slots are not equal, that is, pulling the same arm for different slots gives different results. I do not care about an initial state or feature-vector as the problem always starts from the same setting (it is not contextual). My reward depends on how I pull one of the 7 arms of all of the 66 bandits simultaneously, where, as said, each slots has its own unique properties towards the calculation of the total reward. Basically, the action space is a one-hot encoded 66x7 matrix.
My solution: I ignored the fact that I do not care about a feature vector or state and I treated the problem using a deep NN with a basic policy-gradient algorithm, where I increase directly the probability of each action depending on the reward I get. The state simply does not change, so the NN receive always the same input. This solution does work effectively in finding an approximately optimal strategy, however, it is very computationally expensive and something tells me I am overkilling the problem.
However, I do not see how I could apply standard solutions to MAB such as epsilon-greedy and so on. I need simultaneity between the different "slot-machines" and if I just take each possible permutation as a different action, in order to explore them with greedy methods, I get way too many actions (in the order of 10^12). I have not found in the literature something similar to this multi-armed multi-bandit problem and I am clueless if anything like that has ever been considered - perhaps I am overthinking it and this can be somehow reduced to a normal MAB?