Why do we need function approximation in RL

Learning with tabular methods is much more unstable compared to learning with function approximation.

Because the state and action space may be big or combinatorially enormous, rendering tabular methods impossible to use.

Because we want our agent to be memory-, space- and data efficient.

Relying on function approximation allows us to achieve greater reward in any environment.

———————————————————————————————-

TD targets have small variance.

In TD learning we cannot update the model until the end of an episode is reached.

MC targets have small variance.

In MC learning we cannot update the model until the end of an episode is reached.

In TD learning we can use as few as one step of experience

(s, a, r, s^{'})

to update the model.

In MC learning we can use as few as one step of experience

(s, a, r, s^{'})

to update the model.

———————————————————————————————-

Reward function.

Policy function.

Discount factor

\gamma

Value function.

Expectation of targets.

———————————————————————————————-

MC is online.

MC is offline.

TD is offline.

TD is online.

MC & TD >>> Why do we need function approximation in RL >>> Why do we need function approximation in RL