I think what you have here (with an important caveat, which I will get to later) is a common misunderstanding about how rewards should be structured for a reinforcement learning (RL) problem. It is perfectly OK to return a reward of $0$ for most steps, and only return measured rewards when the event they are related to happens.
The task of figuring out that the critical action that caused a reward was some number of time steps ago is called the credit assignment problem. Sometimes this can make solving an RL problem hard. However, it is a core issue in RL, and all RL algorithms are built to solve this issue. This is usually achieved through algorithms that backup through time iteratively, assigning expected future values to earlier states (in value-based algorithms), or by associating all actions taken with rewards that occured in their future and allowing this to average out over multiple different experiences (in policy gradient methods).
The important caveat is this: When the agent takes the earlier action, this choice must have had some observable influence on the state, and that change should still persist in some form and to some degree at the time the associated reward is observed, such that it is possible for the agent to make the association. In some cases - e.g. a game of checkers - then the state change appears naturally in how the environment is expressed. In other cases you may need to take extra steps. In the worst-case scenario, you would need to store a history of actions taken and add that to the state vector.
Most simple evironments provided with Gym should provide you with a fully usable state vector where all influences on next state and reward are included. So you may need to do nothing other than apply a standard RL control method. Custom, or more advanced environments may require that you add something extra to the state yourself.