A relatively recent but interesting paper that discusses this topic in more detail is Reward is enough (Artificial Intelligence, 2021) by David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton (so by some of the godfathers of RL, who are all at DeepMind).
Their reward-is-enough hypothesis (RIEH) (page 4) is
Hypothesis (Reward-is-Enough). ...
Any possible action, including changing the reward function, would be evaluated through the initial reward function. In order to avoid the scenario you described, a reward function needs to disincentivize changes to itself by giving those the lowest possible reward.