Boosted Tree

Definition:

\hat{y} = \sum_{k = 1}^{K} f_{k} (x)

$\widehat y=\sum_{k=1}^{K}f_k(x)$

In which $f_k(x)$ is one of K regression trees.

Loss:

L o s s = \sum_{i = 1}^{n} L (y_{i}, {\hat{y}}_{i})

$Loss=\sum_{i=1}^{n}L(y_i, \widehat y_i)$

Add some regularization:

L o s s = \sum_{i = 1}^{n} L (y_{i}, {\hat{y}}_{i}) + \sum_{k = 1}^{K} Ω (f_{k})

$Loss=\sum_{i=1}^{n}L(y_i,\widehat y_i) + \sum_{k=1}^{K}\Omega(f_k)$

Additive Training:

{\hat{y}}^{(1)} = 0

$\widehat y^{(1)} = 0$

{\hat{y}}^{(t)} = {\hat{y}}^{(t - 1)} + f_{t} (x)

$\widehat y^{(t)} = \widehat y^{(t-1)} + f_t(x)$

L o s s^{(t)} = \sum_{i = 1}^{n} L (y_{i}, {\hat{y}}_{i}^{(t)}) + \sum_{k = 1}^{t} Ω (f_{k})

$Loss^{(t)}=\sum_{i=1}^{n}L(y_i, \widehat y_i^{(t)}) + \sum_{k=1}^{t}\Omega(f_k)$

= \sum_{i = 1}^{n} L (y_{i}, {\hat{y}}_{i}^{(t - 1)} + f_{t} (x_{i})) + \sum_{k = 1}^{t - 1} Ω (f_{k}) + Ω (f_{t})

$=\sum_{i=1}^{n}L(y_i, \widehat y_i^{(t-1)}+f_t(x_i))+\sum_{k=1}^{t-1}\Omega(f_k)+\Omega(f_t)$

= \sum_{i = 1}^{n} L (y_{i}, {\hat{y}}_{i}^{(t - 1)} + f_{t} (x_{i})) + Ω (f_{t}) + C

$=\sum_{i=1}^{n}L(y_i, \widehat y_i^{(t-1)}+f_t(x_i))+\Omega(f_t)+C$

\approx \sum_{i = 1}^{n} [L (y_{i}, {\hat{y}}_{i}^{(t - 1)}) + f_{t} (x_{i}) \frac{\partial L}{\partial {\hat{y}}_{i}^{(t - 1)}} + \frac{1}{2} f_{t}^{2} (x_{i}) \frac{\partial L^{2}}{\partial {\hat{y}}_{i}^{(t - 1)}}] + Ω (f_{t}) + C

$\approx\sum_{i=1}^{n}[ L(y_i,\widehat y_i^{(t-1)})+f_t(x_i)\frac{\partial L}{\partial \widehat y_i^{(t-1)}}+\frac{1}{2}f_t^{2}(x_i)\frac{\partial L^2}{\partial \widehat y_i^{(t-1)}}]+\Omega(f_t)+C$

= \sum_{i = 1}^{n} [L (y_{i}, {\hat{y}}_{i}^{(t - 1)}) + f_{t} (x_{i}) G_{i} + \frac{1}{2} f_{t}^{2} (x_{i}) H_{i}] + Ω (f_{t}) + C

$=\sum_{i=1}^{n}[ L(y_i,\widehat y_i^{(t-1)})+f_t(x_i)G_i+\frac{1}{2}f_t^{2}(x_i)H_i]+\Omega(f_t)+C$

= \sum_{i = 1}^{n} [f_{t} (x_{i}) G_{i} + \frac{1}{2} f_{t}^{2} (x_{i}) H_{i}] + Ω (f_{t}) + C^{'}

$=\sum_{i=1}^{n}[f_t(x_i)G_i+\frac{1}{2}f_t^{2}(x_i)H_i] + \Omega(f_t) + C'$

Loss at time t is:

L o s s^{(t)} = \sum_{i = 1}^{n} [f_{t} (x_{i}) G_{i} + \frac{1}{2} f_{t}^{2} (x_{i}) H_{i}] + Ω (f_{t}) + C^{'}

$Loss^{(t)}=\sum_{i=1}^{n}[f_t(x_i)G_i+\frac{1}{2}f_t^{2}(x_i)H_i] + \Omega(f_t) + C'$

Use:

f_{t} (x) = w_{q (x)}, q : R^{d} \to {1, 2, . . ., M}, w_{i} \in R

$f_t(x)=w_{q(x)}, q:R^d\rightarrow\{1,2,...,M\}, w_i \in R$

Ω (f) = \frac{1}{2} λ \sum_{i = 1}^{M} w_{j}^{2} + γ M

$\Omega(f)=\frac{1}{2}\lambda\sum_{i=1}^{M}w_j^{2}+\gamma M$

We get:

L o s s^{(t)} = \sum_{i = 1}^{n} [f_{t} (x_{i}) G_{i} + \frac{1}{2} f_{t}^{2} (x_{i}) H_{i}] + Ω (f_{t}) + C^{'}

$Loss^{(t)}=\sum_{i=1}^{n}[f_t(x_i)G_i+\frac{1}{2}f_t^{2}(x_i)H_i] + \Omega(f_t) + C'$

= \sum_{i = 1}^{n} [w_{q (x_{i})} G_{i} + \frac{1}{2} w_{q (x_{i})}^{2} H_{i}] + \frac{1}{2} λ \sum_{j = 1}^{M} w_{j}^{2} + γ M + C^{'}

$=\sum_{i=1}^{n}[w_{q(x_i)}G_i+\frac{1}{2} w_{q(x_i)}^2H_i]+\frac{1}{2}\lambda\sum_{j=1}^{M}w_j^{2}+\gamma M+C'$

With $I_j=\{i|q(x_i)=j\}$ :

\sum_{i = 1}^{n} w_{q (x_{i})} G_{i} = \sum_{j = 1}^{M} [w_{j} \sum_{i \in I_{j}}^{} G_{i}]

$\sum_{i=1}^{n}w_{q(x_i)}G_i=\sum_{j=1}^{M}[w_j\sum_{i\in I_j}^{}G_i]$

\sum_{i = 1}^{n} \frac{1}{2} w_{q (x_{i})}^{2} H_{i} = \sum_{j = 1}^{M} w_{j}^{2} \sum_{i \in I_{j}}^{} \frac{1}{2} H_{i}

$\sum_{i=1}^{n}\frac{1}{2}w_{q(x_i)}^2H_i=\sum_{j=1}^{M}w_j^2\sum_{i \in I_j}^{}\frac{1}{2}H_i$

So:

L o s s^{(t)} = \sum_{j = 1}^{M} [w_{j} \sum_{i \in I_{j}} G_{i} + w_{j}^{2} \sum_{i \in I_{j}} \frac{1}{2} H_{i} + \frac{1}{2} λ w_{j}^{2}] + γ M + C^{'}

$Loss^{(t)}=\sum_{j=1}^{M}[w_j\sum_{i\in I_j}G_i+w_j^2\sum_{i\in I_j}\frac{1}{2}H_i+\frac{1}{2}\lambda w_j^2]+\gamma M + C'$

= \sum_{j = 1}^{M} [w_{j} \sum_{i \in I_{j}} G_{i} + \frac{1}{2} w_{j}^{2} (λ + \sum_{i \in I_{j}} H_{i})] + γ M + C^{'}

$=\sum_{j=1}^{M}[w_j\sum_{i\in I_j}G_i+\frac{1}{2}w_j^2(\lambda+\sum_{i\in I_j}H_i)]+\gamma M + C'$

With $G_j'=\sum_{i\in I_j}G_i, H_j'=\sum_{i\in I_j}H_i$ :

L o s s^{(t)} = \sum_{j = 1}^{M} [w_{j} G_{j}^{'} + \frac{1}{2} w_{j}^{2} (λ + H_{j}^{'})] + γ M + C^{'}

$Loss^{(t)}=\sum_{j=1}^{M}[w_jG_j'+\frac{1}{2}w_j^2(\lambda+H_j')]+\gamma M + C'$

Finally:

w_{j}^{*} = a r g m i n (w_{j} G_{j}^{'} + \frac{1}{2} w_{j}^{2} (λ + H_{i}^{'})) = - \frac{G_{j}^{'}}{λ + H_{i}^{'}}

$w_j^*=argmin(w_jG_j'+\frac{1}{2}w_j^2(\lambda+H_i'))=-\frac{G_j'}{\lambda+H_i'}$

O b j^{(t)} = m i n (L o s s^{(t)}) = - \frac{1}{2} \sum_{j = 1}^{M} \frac{G_{j}^{' 2}}{H_{j}^{'} + λ} + γ M + C^{'}

$Obj^{(t)}=min(Loss^{(t)})=-\frac{1}{2}\sum_{j=1}^{M}\frac{G_j'^2}{H_j'+\lambda}+\gamma M + C'$

So for each iteration t of training, greedily seach for a regression tree $f_t(x_i)=w_{q(x_i)}$ with $w_j=-\frac{G_j'}{\lambda+H_i'}$ with minumum $Obj^{(t)}$ and add it to model.

猜你喜欢