Deep learning Adam optimization algorithm theoretical knowledge and changes in learning rate

In recent learning, Adam was used as the optimizer and the learning rate was printed during training and it was found that the learning rate did not change. This seems to contradict the previously understood adaptive learning rate?

Adam's theoretical knowledge

Adam paper: https://arxiv.org/pdf/1412.6980.pdf
Insert image description here The picture above is the detailed process of applying the Adam optimization algorithm to the gradient descent method in deep learning. There are some parameters that need to be explained: the
details can be found through https://blog .csdn.net/sinat_36618660/article/details/100026261 to understand Adam’s principle.

Question 1 What is an exponential moving average?

Exponential Moving Average (EMA) Exponential Moving Average means that the weighting coefficient of each value decreases exponentially with time. The closer the value is to the current moment, the greater the weighting coefficient.
Insert image description here
Take $m_t$ For example, from the above derivation, we can see that the farther away from the time t, the smaller the proportion of the gradient. In the process of continuous updating of the gradient, although the historical gradient is used, the gradient at different moments has a greater impact on the current $m_t$ The contributions are different. The closer to the time t, the greater the contribution to $m_t$ The greater the influence, the further away from time t, the greater the impact on $m_t$ The impact will be smaller.

Question 2 Why does it need to be corrected?

(1) Popular explanation:

When $m_t$ Without correction ( $\beta_1=0.9$ )：
$m_0$ =0
$m_1$ = $\beta _1* m_0+(1-\beta_1)*g_1$ = $0.1g_1$
$m_2$ = $0.9*0.1*g_1+0.1*g_2$ = $0.09g_1+0.1g_2$
$m_3$ = $0.081g_1+0.09g_2+0.1g3$
and so on, we can see that since $m_0$ =0, resulting in, $m_t$ are biased towards 0 and will also move away from $g_t$ further and further.

(2) Explanation of theoretical formula:

$m_t$ from above $m_{t}$ It can be seen from the update formula that $m_t$ Equivalent to gradient $g_t$ First-order distance estimate, so we calculate $m_t$ Expectation:
Insert image description here
As can be seen from the above formula, $m_t needs to be$ Corrected to $m_t/(1-\beta_1^t)$ , $m_t$ is the gradient $g_t$ unbiased estimate of . The same idea can explain $v_t$ correction.

Question 3 How does the learning rate change?

In Adam's paper, it is pointed out that the following three lines of formulas can be
Insert image description here
replaced by: Equivalently replaced by:

In the pytorch source code, it is also written in the above way (attached at the end).

Does writing like this mean that the change of learning rate is determined by $\alpha \sqrt{1-\beta_2^t}/(1-\beta_1^t)$ to decide, can this still be called an adaptive learning rate?

We know the definition formula of the gradient descent method:
Insert image description here
According to the definition formula of the gradient descent method, we can write the parameter update formula as:

$\theta_t=\theta_{t-1}- {\frac{\alpha}{\sqrt{\hat{v_t}}+\varepsilon}}*\hat{m_t}$
Among them, $m_t$ Considered as gradient $g_t$ The first-order distance estimate of , then ${\frac{\alpha}{\sqrt{\hat{v_t}}+\varepsilon}}$ It can be seen that at time t, the parameter $\theta_t$ learning rate. It can also be seen from the above formula that for different parameters, there will be different learning rates at each moment, so it is difficult to visualize them.

Finally, the Adam source code is attached

I found the adam.py file through pytorch1.2/lib/python3.7/site-packages/torch/optim/, the following is the code:

def step(self, closure=None):
  loss = None
  if closure is not None:
      loss = closure()

  for group in self.param_groups:
      for p in group['params']:
          if p.grad is None:
              continue
          grad = p.grad.data
          if grad.is_sparse:
              raise RuntimeError('Adam does not support sparse gradients, please consider SparseAdam instead')
          amsgrad = group['amsgrad']

          state = self.state[p]

          # State initialization
          if len(state) == 0:
              state['step'] = 0
              # Exponential moving average of gradient values
              state['exp_avg'] = torch.zeros_like(p.data)
              # Exponential moving average of squared gradient values
              state['exp_avg_sq'] = torch.zeros_like(p.data)
              if amsgrad:
                  # Maintains max of all exp. moving avg. of sq. grad. values
                  state['max_exp_avg_sq'] = torch.zeros_like(p.data)

          exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
          if amsgrad:
              max_exp_avg_sq = state['max_exp_avg_sq']
          beta1, beta2 = group['betas']

          state['step'] += 1

          if group['weight_decay'] != 0:
              grad.add_(group['weight_decay'], p.data)

          # Decay the first and second moment running average coefficient
          exp_avg.mul_(beta1).add_(1 - beta1, grad)
          exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
          if amsgrad:
              # Maintains the maximum of all 2nd moment running avg. till now
              torch.max(max_exp_avg_sq, exp_avg_sq, out=max_exp_avg_sq)
              # Use the max. for normalizing running avg. of gradient
              denom = max_exp_avg_sq.sqrt().add_(group['eps'])
          else:
              denom = exp_avg_sq.sqrt().add_(group['eps'])

          bias_correction1 = 1 - beta1 ** state['step']
          bias_correction2 = 1 - beta2 ** state['step']

          step_size = group['lr'] * math.sqrt(bias_correction2) / bias_correction1

          p.data.addcdiv_(-step_size, exp_avg, denom)

  return loss

The above is my personal understanding. If you have better ideas, please leave a message and learn and progress together!