In Barto and Sutton's book, it's written that we have two types of updates in dynamic programming
- Update out-of-place
- Update in-place
The update in-place is the faster one. Why is that the case?
This is the pseudocode that I used to test it.
if in_place: state_values = new_state_values else: state_values = new_state_values.copy() old_state_values = state_values.copy() for i in range(WORLD_SIZE): for j in range(WORLD_SIZE): value = 0 for action in ACTIONS: (next_i, next_j), reward = step([i, j], action) value += ACTION_PROB * (reward + discount * state_values[next_i, next_j]) new_state_values[i, j] = value max_delta_value = abs(old_state_values - new_state_values).max() if max_delta_value < 1e-4: break
Why is the in-place version faster, and what is the difference? What I think is that it is only better for storage usage, I don't understand the speed increase part.