I have a simulator modelling a relatively complex scenario. I extract ~12 discrete features from the simulator state which forms the basis for my MDP state space.
Suppose I am estimating the transition table for an MDP by running large number of simulations and extracting feature transitions as the state transitions.
While I can randomize the simulator starting conditions to increase the coverage of states, I cannot guarantee all states will be represented in the sample ie. states which are possible but rare.
Is there a rigorous approach to "filling in the gaps" of the transition table in this case?
1) For each state which was unrepresented in the sample, simply transition to all other states with equal probability, as a "neutral" way to fill in the gap?
2) As above, but transition only to represented states (with equal probability)?
3) Transition to same state with probability 1.0?
4) Ignore unrepresented states during MDP solving entirely, and simply have a default action specified?