2.1 Markov Reward Processes

An agent operatingin aMarkovian environment transitionsfrom state to state,ingeneral obtaining rewards along the way, as follows: at time t,

1. state is st
2. receive reward rt
3. transition to state st+1 with probability P[st+1 | st]

We modelthis agent’sinteractions as a(discrete-time) Markov rewardprocess, a tuple �S,R,P�, where time is discrete: i.e., t ∈ T = {0,1,...}, and

• S is a finite set of states
• R : S → R is a reward function
• P : S → Δ(S)is aprobability transitionfunction(or matrix) Δ(S)is the set of probability distributions over S

Theform oftheprobabilitytransitionfunction encodes thefact thatitsatisfies theMarkovproperty.

Remark: Markov reward processes can have stochastic rewards as well as stochastic transitions. But our framework is sufficiently general, since such processes can be reduced to Markov reward processes withdeterministic rewards simplybylettingthedeterministic rewards equal the expected values of the corresponding stochastic rewards.

Example: Gambler’s Ruin is an example of aMarkov rewardprocess. Agamblergambles until he either wins a set amount of money, say $N, or loses all his money. At state st, his wealth increases by $1 with probability 1/3, and it decreases by $1 with probability 2/3.

10:30 AM, Apr 7, 2009

The set of states is defined by the worth of the gambler: S = {0,. . .,N, end}. The rewards are defined as R(end)= R(i)= 0, for i =0,...,N − 1, but R(N)= 1. The transition probabilities are such that P[i +1 | i] =1/3 and P[i − 1 | i] =2/3, for i =1,...,N − 1; P[end | i] = 1, for i =0,N; and P[end | end]=1.

Given initial state i ∈ 1,...,N − 1, what is the probability that the gambler wins: i.e., reaches state N?

Figure 1: Gambler’s Ruin: N = 4. An absorbing state s ∈ S is s.t. P[s | s]= 1. The end state is an absorbing state in Gambler’s Ruin.

3.1 Return

Given trajectory τ = ...), the returnρ^τ at time t is a function of the current reward

^(st^,st+1^,st+2^,t rt and the stream of future rewards rt+1,rt+2,.... In the case of a finite horizon, say of length T< ∞, return can be computed simply as the sum of current and future rewards: i.e.,

T−t

_ρ^τ

= rt +rt+1 + rt+2 + ... +rT = rt+i (3)

t i=0

In the case of infinite horizons, however, the sum of future rewards is potentially infinite. If all trajectories areproper(atrajectory iscalled a proper trajectory iff it transitions to a zero-reward, absorbing state with positive probability), return can be computed simply as the sum of current andfuturerewards, asinEquation3. Otherwise,returnis computed asthe sumof currentrewards

10:30 AM, Apr 7, 2009

and the discounted sum of future rewards: i.e., assuming discount factor 0 ≤ γ< 1,

∞

ρ^τ = rt + γrt+1 + γ² rt+2 + ... = γⁱ rt+i (4)

t i=0

If rewards are assumed to be bounded, return as defined by Equatio n 4 is finite. (Why?) But even inthecaseof finitehorizonsorpropertrajectories, ρ^τ is often computed with discounting, because

t

of the following economic “law”: a dollar today is worth more than a dollar tomorrow.

Thislaw encapsulatesthefollowing equation, which statesthatthefuturevalue(FV) of money equalsthepresent value(PV) scaledby theinterest rate(r):

FV =PV(1+r)

For example, $d today wouldbeworth$(1+ r)d 1 year from now. Similarly, $d that are scheduled to be accrued 1 year from now are worth only $d/(1+r)today.

The discount factor γ is inversely related to the interest rate: γ =1/(1+ r). Intuitively, γ determines the relative worth of immediate vs. future rewards. As γ → 0, immediate rewards are deemed more and more relevant; agents that attempt to maximize return in these circumstances are called myopic. As γ → 1,futurerewardsareweighted more and moreheavily; agentsthat aim to maximize return in these circumstances exhibit foresight.

3.2 Bellman’s Theorem

We can now stateBellman’s seminaltheoremforMarkov rewardprocesses(i.e.,for state values).

Theorem: The state value V(st) at state st—which is defined as the expected reward that is accrued from time t on—can be equivalently expressed as the sum of the immediate reward rt and the discounted expected value of state st+1: i.e., for 0 ≤ γ< 1,

V(st)= rt + γE[V(st+1)] (5)

′ ′′

Proof: (Sketch)In what follows, τ =(st+1,st+2,...)and τ =(st+2,...).

V(st)= P[τ | st]ρ^τ

t τ

′

= rt + γP[τ | st]ρ^{τ ′}

t+1

′

τ

′′

= rt + γP[τ | st+1,st]ρ^{τ ′′}

P[st+1 | st] rt+1 + γ t+2

′′

st+1∈Sτ

′′

′′

= rt + γP[st+1 | st] rt+1 + γP[τ | st+1]ρ_t^τ ₊₂

′′

st+1∈Sτ

′

= rt + γP[τ ^′

P[st+1 | st] | st+1]ρ^τt+1

′

st+1∈Sτ

= rt + γP[st+1 | st]V(st+1)

st+1∈S

= rt + γE[V(st+1)]

10:30 AM, Apr 7, 2009

The fourth line follows from the Markov property. The seventh line is simply an abbreviation.

3.3 Bellman’s Equations

Bellmans’ theoremgives risetothefollowing system of |S| equations with |S| unknowns, known as Bellman’s equations: for all states s ∈ S,

′

V(s)= R(s)+γP[s | s]V(s ^′ ) (6)

s ′

To find asolutiontothissystemof equations,werely onBanach’s fixedpointtheorem,alsocalled the contraction mapping theorem. Given a metric space ¹ (X,d), a mappingf : X → X is called a contraction iff there exists some 0 ≤ k< 1 s.t. d(f(x),f(y))≤ kd(x,y), for allx,y ∈ X.

The L∞, or max, norm is defined as follows on Rⁿ: for all x,y ∈ Rⁿ , ||x− y|| = max1≤i≤n |xi − yi|. In the special case where n =1, the max norm reduces to absolute value.

1

Example: The function f(x)= x is a contraction mapping on(R,L∞), since |f(x)− f(y)| =

�� 2

�1 � 1

� x − ¹ y� = |x − y|.

22 2

Banach’s Theorem: Given a complete ² metric space (X,d) as well as a contraction mapping

∗∗ 0

f : X → X,(i) there exists a unique x ∈ X s.t. f(x ^∗)= x ; and(ii) forarbitrary x∈ X, the n+1 _fn+1_(x∗

sequence {xⁿ} defined by x= f(xⁿ)= ⁰)converges to x .

Define the mapping f : R^S → R^S as follows:

′

(f(x))(s)= R(s)+γP[s | s]x(s ^′ ) (7)

s ′

Theorem: The mapping f inEquation 7 is a contraction on(R^S,L∞).

Proof: For all x,y ∈ X, and for arbitrary state s ∈ S,

|(f(x))(s)− (f(y))(s)|

� ′′ �

= � R(s)+γP[s | s]x(s ^′ )− R(s)+γP[s | s]y(s ^′ ) �

s ′ s ′

′

= γP[s | s]^�x(s ^′ )− y(s ^′ )

s ′

′ ′′ _{)− y(s} ′′ _)|

≤ γP[s | s]max |x(s

_s ′′ s ′

′

= γP[s | s]||x − y||

s ′

= γ||x − y||

It follows that |(f(x))(s)− (f(y))(s)|≤ γ||x − y||, for all states s. Therefore, ||f(x)− f(y)|| = maxs |(f(x))(s)− (f(y))(s)|≤ γ||x − y||.

Corollary: Bellman’ssystemof equations(Equation 6) indeedhasa fixedpointsolution,and the iterative application of f converges to this solution.

¹A metric space(X, d)is a set X together with a distance function d : X × X → R that satisfies: (i) d(x, x)=0, for all x ∈ X;(ii) d(x, y)= d(y, x), for all x, y ∈ X; and(iii) thetriangleinequality—d(x, z)≤ d(x, y)+d(y, z), for all x, y, z ∈ X.

²An example of a complete metric space is R.

10:30 AM, Apr 7, 2009