20 lines
881 B
Markdown
20 lines
881 B
Markdown
Let's define the Bellman Optimality Equation: $v(s)=$ $( v^*(s) = \max_a \sum_{s', r} p(s',r|s,a)[r + \gamma v^*(s')])$, where $(v^*(s))$ represents the optimal value function for state \(s\).
|
|
|
|
The equation is not closed, but there are several ways to compute it:
|
|
- dynamic programming
|
|
- ...
|
|
- Q-Learning
|
|
|
|
Q-Learning is an iterative way to do it, that learns the optimal values "online".
|
|
|
|
Q-Learning is the fusion of TD-Learning and Off Policy.
|
|
|
|
#### Temporal Difference Learning
|
|
At each step the state value is updates:
|
|
$$V(S_{t})=V(S_{t})+\alpha[R_{t+1}+\gamma V(S_{t+1})-V(S_{t})]$$
|
|
|
|
This is a special case of the $TD(\lambda)$ called $TD(0)$, or one-step TD. It works by updating the previous estimate every time an action has been taken.
|
|
|
|
#### Q-Learning
|
|
$$Q(S, a)=Q(S, a)+\alpha(R+\gamma max_{a'}(Q(S', a')-Q(S, a))$$
|
|
This will converge to the optimal action value function.
|