Shie Mannor, Yishay Mansour, Aviv Tamar
This chapter provides essential background on Markov chains as the foundation for Markov decision processes. The theory of state classification is developed, introducing accessibility relations, communicating classes and irreducibility. States are classified as recurrent or transient. The concept of periodicity is introduced. Central results concern invariant distributions – probability distributions that remain unchanged under the chain’s dynamics. Conditions for the existence and uniqueness of stationary distributions are established.