What are the two main steps in value-based approach to Reinforcement Learning

2. Extract a reward function from the value function

2. Extract a value function from the reward function

2. Extract a value function from policy

1. Build a value function

1. Estimate a reward function

2. Extract a policy function from the value function

1. Build a policy function

—————————————————————————————–

If the policy improvement step computes the new policy

\pi’

based on the estimates of the expected return

V_\pi(s)

for the old policy

\pi

, the new policy might have worseexpected return

V_{\pi’}(s)

in some states. This is so because the estimates will no longer be valid as soon as the policy is changed (improved).

Repeating the policy improvement step several times in a row may benefit the performance of the resulting policy.

An agent acts greedily with respect to a combination of the immediate reward and the expected return under policy

\pi

An agent acts greedily with respect to the immediate reward only and ignores the remaining expected return under policy

\pi

—————————————————————————————–

Infinitely many.

Depends on the number of actions.

Depends on the number of states.

One.

—————————————————————————————–

The system of Bellman equations may have no solution at all. Thus we should be satisfied with an approximation.

The solution of such system of equations is intractable on any modern supercomputer. Thus we have to approximate.

After reaching some precision level further refinements of the solution will not change the result of subsequent policy improvement.

We want to sacrifice the global optimality for much faster convergence.

—————————————————————————————–

does not require to perform policy evaluation until its convergence

requires to perform policy evaluation until convergence at every iteration.

does not depend on initialization.

depends on initialization.

converges to a global optimum.

converges to a local optimum.

requires to improve policy in each and every state before subsequent policy evaluation.

does not require to improve policy in each and every state as long as policy in any state is improved once in a while

—————————————————————————————–

Using the

a r g m a x

operator.

Find the action whose

q

-value is closest to the average of

q

-values over actions.

It is impossible without the knowledge of environment dynamics

\mid s, a)

Sample from a distribution that is proportional to

q

-values.

Using the

\max

operator.

—————————————————————————————–

Policy Iteration updates value function

V_\pi

until numerical convergence of all its state values

V_\pi(s)

before each policy improvement step.

Value Iteration updates value function

V

until numerical convergence of all its state values

V (s)

before each policy improvement step.

Policy Iteration performs only one iteration of policy evaluation before each policy improvement step.

Value Iteration performs only one iteration of policy evaluation before each policy improvement step.

Home » Practical Reinforcement Learning » What are the two main steps in value-based approach to Reinforcement Learning

What are the two main steps in value-based approach to Reinforcement Learning

Policy Iteration >>> What are the two main steps in value-based approach to Reinforcement Learning >>> Practical Reinforcement Learning

1.

2.

3.

4.

5.

6.

7.

What are the main sources of randomness in reinforcement learning

In broad strokes how do policy based methods work

How is a model-free RL algorithm different from a model-based one

What is true about Bellman equations

Why do we need function approximation in RL

Which of the following is true about regret