（1）深度强化学习基础【基本概念】

1. Terminologies

There are many professional terms in reinforcement learning. If you want to get started with reinforcement learning, you must understand these professional terms.

1] state and action
state $s$ (this frame)

action $a$ $\in$ {left, right, up}

Who does the action is the agent.

2] policy

policy $\pi$ : According to the observed state, make decisions and control the movement of the agent.

$\cdot$ Policy function $\pi$ : $(s,a)$ → [0, 1] :
$\pi (a|s)=P(A=a|S=s).$
$\cdot$It is the probability of taking action $A=a$ given $s$, e.g,
$\cdot \pi (left|s)=0.2,$
$\cdot \pi (right|s)=0.1,$
$\cdot \pi (up|s)=\mathrm{0.7.}$
$\cdot$ Upon observing state $S=s$, the agent’s action A can be random.

3] reward

reward $R$
$\cdot$ Collect a coin: $R$ = +1
$\cdot$ Win the game: $R$ = +10000
$\cdot$ Touch a Goomba: $R$ = -10000
       \;\;\; (game over)
$\cdot$ Nothing happens: $R$ = +1

4] state transition

         \;\;\;\; old state$\stackrel{action}{⟶}$new state
$\cdot$ E.g., “up” action leads to a new state.
$\cdot$ State transition can be random.
$\cdot$ Ramdom is from the environment.
$\cdot$ $p(s^{{}^{\prime }}|s,a)$=$P(S^{{}^{\prime }}=s|S=s,A=a).$

5] agent environment interaction

The environment here is a game program, the agent is Mary, and the state $s_t$ is what the environment tells us. In super Mary, we can take the current picture as the environment $s_t$. when we see the state st, we need to make an action $a_t$, which can be left, up, right.

After making the action $a_t$, we will get a new state and a reward $r_t$.

2. Randomness in Reinforcement Learning

1] Action have randomness

$\cdot$ Given state $s$, the action can be random, e.g,.
         \;\;\;\; $\cdot$ $\pi ("left|s")=0.2$
         \;\;\;\; $\cdot$ $\pi ("right|s")=0.1$
         \;\;\;\; $\cdot$ $\pi ("up|s")=0.7$

Actions are sampled by pocily function.

2] State transitions have randomness

$\cdot$ Given state $S=s$ and action $A=a$, the environment randomly generates a new state $S^{{}^{\prime }}$.

The new state is sampled by the state transition function.

3. Play the game using AI

$\cdot$ Observe a frame(state $s_1$)
$\cdot$$\Rightarrow$ Make action $a_1$ (left, right, or up)
$\cdot$$\Rightarrow$Observe a new frame(state $s_2$) and reward $r_1$
$\cdot$$\Rightarrow$ Make action $a_2$
$\cdot$$\Rightarrow$ …

$\cdot$ (state, action, reward) trajectory:
$s_1,a_1,r_1,s_2,a_2,r_2,......,s_T,a_T,r_T.$

4. Rewards and Returns (important)

4.1 Rerun

Definition: Return (cumulative future reward)

$\cdot$ $U_t=R_t+R_{t+1}+R_{t+2}+R_{t+3}+...$

Question: Are $R_t$ and $R_{t+1}$ equally important?
$\cdot$ Which of the followings do you prefer?
       \;\;\; $\cdot$ I give you $100 right now.
       \;\;\; $\cdot$ I will give you $100 one year later.
$\cdot$ Future reward is less valuable than present reward.
$\cdot$ $R_{t+1}$ should be given less weight than $R_t$

Definition: Discounted return(cumulative discounted future reward)

$\cdot$ $\gamma$: discount rate (tuning hyper-parameter).

$\cdot$ $U_t=R_t+\gamma R_{t+1}+{\gamma }^2{R}_{t+2}+{\gamma }^3{R}_{t+3}+...$

4.2 Randomness in Returns

Definition: Discounted return(at time step t)

$\cdot$ $U_t=R_t+R_{t+1}+R_{t+2}+R_{t+3}+...$

At time step t, the return $U_t$ is random.
$\cdot$ Two sources of randomness:
         \;\;\;\; 1. Action can be random: $P[A=a|S=s]=\pi (a|s).$
         \;\;\;\; 2. New state can be random: $P[S^{{}^{\prime }}=s|S=s,A=a]=p(s^{{}^{\prime }}|s,a).$

$\cdot$ For any i $\ge$ t, the reward $R_i$ depends on $S_i$ and $A_i$.

$\cdot$ Thus, given $s_t$, the return $U_t$ depends on the random variables:
       \;\;\; $\cdot$ $A_t,A_{t+1},A_{t+2},...$ and $S_{t+1},S_{t+2},...$

5. Value Function

5.1 Action-Value Function $Q(s,a)$

Definition: Return (cumulative future reward)
$\cdot$ $U_t=R_t+R_{t+1}+R_{t+2}+R_{t+3}+...$

Definition: Action-value function for policy $\pi$
$\cdot$ $Q_{\pi }(s_t,a_t)=E[U_t|S_t=s_t,A_t=a_t]$

$\cdot$ Return $U_t$ (random variable) depends on actions $A_t,A_{t+1},A_{t+2},...$ and $S_t,S_{t+1},S_{t+2},...$

$\cdot$ Actions are random: $P[A=a|S=s]=\pi (a|s).$ (Policy function)

$\cdot$ States are random: $P[S^{{}^{\prime }}=s|S=s,A=a]=p(s^{{}^{\prime }}|s,a).$ (State transition)

Action value function represents: if the policy function is used $\pi$, then whether it is good or bad to act $a_t$ in the state of $s_t$, we know the policy function $\pi$, You can score all actions $a$ in the current state.

Definition: Optimal action-value function
$\cdot$ $Q_{\pi }^{\ast }(s_t,a_t)=\underset{\pi }{\max }$ $Q_{\pi }^{}(s_t,a_t)$
Evaluate action $a$ to tell the best action.

5.2 State-Value Function $V(s)$

Definition: State-value function
$\cdot$ $V_{\pi }(s_t)=$ $E_A[Q_{\pi }^{}(s_t,A)]={\sum }_a\pi (a|s_t)\cdot Q_{\pi }(s_t,a)$. (Actions are discrete)

$\cdot$ $V_{\pi }(s_t)=$ $E_A[Q_{\pi }^{}(s_t,A)]=\int \pi (a|s_t)\cdot Q_{\pi }(s_t,a)da$. (Actions are continuous)

$V_{\pi }(s_t)$ could make a judgment on the current situation and tell us whether we are going to win or lose, or others.

5.3 Understanding the Value Functions

$\cdot$ Action-value function: $Q_{\pi }(s_t,a_t)=E[U_t|S_t=s_t,A_t=a_t]$.
$\cdot$ For policy $\pi$, $Q_{\pi }(s,a)$ evaluates how good it is for an agent to pick action $a$ while being in state $s$.

$\cdot$ State-value function: $V_{\pi }(s_t)=E_A[Q_{\pi }^{}(s_t,A)]$
$\cdot$ For fixed policy $\pi$, $V_{\pi }(s)$ evaluates how good the situation is in state $s$.
$\cdot$ $E_s[V_{\pi }(s)]$ evaluates how good the policy $\pi$ is.

6. How does AI control the agent?

Suppose we have a good policy $\pi (a|s)$.
$\cdot$ Upon observe the state $s_s,$
$\cdot$ random sampling: $a_t\backsim \pi (\cdot |s_t)$.

Suppose we know the optimal action-value function $Q^{\ast }(s,a)$.
$\cdot$ Upon observe the state $s_t,$
$\cdot$ choose the action that maximizes the values: $a_t=argma{x}_a{Q}^{\ast }(s_t,a).$

7. Summary

Agent, Environment, State $s$, Action$a$, Reward$r$, Policy $\pi (a|s)$, State transition $p(s^{{}^{\prime }}|s,a)$.

Return: $U_t=R_t+\gamma R_{t+1}+{\gamma }^2{R}_{t+2}+{\gamma }^3{R}_{t+3}+...$

Action-value function: $Q_{\pi }(s_t,a_t)=E[U_t|S_t=s_t,A_t=a_t]$.

Optimal action-value function: $Q_{\pi }^{\ast }(s_t,a_t)=\underset{\pi }{\max }$ $Q_{\pi }^{}(s_t,a_t)$.

State-value function: $V_{\pi }(s_t)=$ $E_A[Q_{\pi }^{}(s_t,A)]$