Markov Chains

Markov chains have a special property that probabilities indicating how the process will evolve in the future depending only on the present state of the process, and so are independent of events in the past.

In the continuous case, it can be looked at the inventory level of a product in a store at any time while in the discrete case inventory level is observed only during each minute, once a month or at similar intervals.

A stochastic process is a family of random variables
{X(t),t ∈T} defined on a given probability space, indexed by the parameter t, where t varies over an index set T.

In order to be able to analyze a stochastic process, it is needed to make some assumptions on the dependence between the random variables.

The most common dependence structure is called Markov Property.

A Markov chain is described in terms of its transition probabilities and the key assumption underlying a Markov chain is that the stochastic process is a Markov chain, which has the Markov property.

The event will occur given the knowledge that an event A has already occurred.

A convenient way of showing all the transition probabilities is the transition probability matrix, P, which is simply a two-dimensional array whose element at the ith row and jth column is pij

P^k gives the probabilities of a transition from one state to another in k repetitions.

It is the probability that a process in state j will be in state i after n additional transitions. 

A transition matrix is called regular if some power of the matrix includes all positive entries.

Markov chain is a regular Markov chain if its transition matrix is regular.

States i and j are in the same communicating class if each state is accessible from the other, i.e., i↔j.

Markov chain has more than one class, it is called reducible.

A state is called periodic, if it can only return to itself after a fixed number of transitions greater than 1.

In fact, some of the most important applications of Markov chains involve an important class of Markov chains which is called an absorbing chain. A state i of a Markov chain is called an absorbing state if, once the Markov chain enters the state, it locks in there forever. This means that the probability of leaving the state is zero.

The main goal of a Markov chain is to calculate the probability that a system occupies a given state S_i , when n is very large. This probability is called the limiting probability and it may converge to steady-state values that are independent of the initial state.

Markov chain means that the chain does not stop changing but that enough time has elapsed, so that the probabilities do not change with respect to time.

Brand switching is the process of choosing to switch from routine use of one brand or product to steady use of a different but similar brand or product.

Alexander Pushkin