reinforcement learning constant states

Question

I am wondering does it make sense to incorporate constant states in the Markov Decision Process and employ reinforcement learning algorithm to solve it.

For instance,