UA MATH571A 多元线性回归IV 广义线性模型
Y 1 , Y 2 , . . . , Y N Y_1,Y_2,...,Y_N Y 1 , Y 2 , . . . , Y N E Y i = μ i EY_i=\mu_i E Y i = μ i g g g g ( μ i ) g(\mu_i) g ( μ i ) g ( μ i ) = X i β
g(\mu_i) = X_i \beta
g ( μ i ) = X i β g ( μ i ) = μ i g(\mu_i)=\mu_i g ( μ i ) = μ i g g g
回归模型Y i = X i β + ϵ i
Y_i = X_i \beta + \epsilon_i
Y i = X i β + ϵ i Y i = 0 , 1 Y_i = 0,1 Y i = 0 , 1 Y i Y_i Y i p i = P ( Y i = 1 ) = E [ Y i ] = X i β ^
p_i =P(Y_i = 1)= E[Y_i] = X_i \hat{\beta}
p i = P ( Y i = 1 ) = E [ Y i ] = X i β ^
残差项不满足正态假设Y i = 0 Y_i=0 Y i = 0 ϵ i = − X i β ^ \epsilon_i = -X_i \hat{\beta} ϵ i = − X i β ^ Y i = 1 Y_i=1 Y i = 1 ϵ i = 1 − X i β ^ \epsilon_i = 1-X_i \hat{\beta} ϵ i = 1 − X i β ^
同方差假设不成立Y i Y_i Y i σ 2 ( ϵ i ) = − X i β ^ ( 1 − X i β ^ ) \sigma^2(\epsilon_i)=-X_i \hat{\beta}(1-X_i \hat{\beta}) σ 2 ( ϵ i ) = − X i β ^ ( 1 − X i β ^ ) X i X_i X i
回归方程取值受限E [ Y i ] = X i β ^ ∈ [ 0 , 1 ] E[Y_i] = X_i \hat{\beta} \in [0,1] E [ Y i ] = X i β ^ ∈ [ 0 , 1 ]
所以二值被解释变量一般做如下处理。假设Y i c Y_i^c Y i c Y i = 1 , i f Y i c ≤ Y 0 Y i = 0 , i f Y i c > Y 0
Y_i = 1,\ if\ Y_i^c\le Y_0 \\Y_i = 0,\ if\ Y_i^c > Y_0
Y i = 1 , i f Y i c ≤ Y 0 Y i = 0 , i f Y i c > Y 0 P ( Y i = 1 ) = P ( Y i c ≤ Y 0 )
P(Y_i=1) = P(Y_i^c \le Y_0)
P ( Y i = 1 ) = P ( Y i c ≤ Y 0 ) Y i c Y_i^c Y i c Y i c = X i β + ϵ i , ϵ i ∼ N ( 0 , σ c 2 )
Y_i^c = X_i \beta + \epsilon_i,\ \epsilon_i \sim N(0,\sigma^2_c)
Y i c = X i β + ϵ i , ϵ i ∼ N ( 0 , σ c 2 ) P ( Y i = 1 ) = P ( Y i c ≤ Y 0 ) = P ( X i β + ϵ i ≤ Y 0 ) = P ( ϵ i ≤ Y 0 − X i β ) = P ( ϵ i σ c ≤ Y 0 σ c − X i β σ c )
P(Y_i=1) = P(Y_i^c \le Y_0) = P(X_i \beta + \epsilon_i \le Y_0 )\\
=P(\epsilon_i \le Y_0 - X_i \beta) = P(\frac{\epsilon_i}{\sigma_c} \le \frac{Y_0}{\sigma_c} - X_i \frac{\beta}{\sigma_c})
P ( Y i = 1 ) = P ( Y i c ≤ Y 0 ) = P ( X i β + ϵ i ≤ Y 0 ) = P ( ϵ i ≤ Y 0 − X i β ) = P ( σ c ϵ i ≤ σ c Y 0 − X i σ c β )
用Φ \Phi Φ P ( Y i = 1 ) = P ( ϵ i σ c ≤ Y 0 σ c − X i β σ c ) = Φ ( X i β ∗ )
P(Y_i=1) = P(\frac{\epsilon_i}{\sigma_c} \le \frac{Y_0}{\sigma_c} - X_i \frac{\beta}{\sigma_c}) = \Phi(X_i \beta^*)
P ( Y i = 1 ) = P ( σ c ϵ i ≤ σ c Y 0 − X i σ c β ) = Φ ( X i β ∗ ) β 0 ∗ = Y 0 σ c − β 0 σ c β i ∗ = − β i σ c
\beta_0^* = \frac{Y_0}{\sigma_c} - \frac{\beta_0}{\sigma_c} \\
\beta_i^* = - \frac{\beta_i}{\sigma_c}
β 0 ∗ = σ c Y 0 − σ c β 0 β i ∗ = − σ c β i
Logit模型又叫Logistics回归。对于P ( Y i = 1 ) = P ( ϵ i σ c ≤ Y 0 σ c − X i β σ c ) = P ( ϵ i σ c ≤ X i β ∗ )
P(Y_i=1) = P(\frac{\epsilon_i}{\sigma_c} \le \frac{Y_0}{\sigma_c} - X_i \frac{\beta}{\sigma_c}) = P(\frac{\epsilon_i}{\sigma_c} \le X_i \beta^*)
P ( Y i = 1 ) = P ( σ c ϵ i ≤ σ c Y 0 − X i σ c β ) = P ( σ c ϵ i ≤ X i β ∗ ) ϵ L = π 3 ϵ i σ c , β = π 3 β ∗
\epsilon_L = \frac{\pi}{\sqrt{3}} \frac{\epsilon_i}{\sigma_c}, \beta = \frac{\pi}{\sqrt{3}} \beta^*
ϵ L = 3 π σ c ϵ i , β = 3 π β ∗ π 3 \frac{\pi}{\sqrt{3}} 3 π ϵ L \epsilon_L ϵ L F ( ϵ L ) = exp ( ϵ L ) 1 + exp ( ϵ L )
F(\epsilon_L) = \frac{\exp{(\epsilon_L)}}{1+\exp{(\epsilon_L)}}
F ( ϵ L ) = 1 + exp ( ϵ L ) exp ( ϵ L ) P ( Y i = 1 ) = P ( ϵ i σ c ≤ X i β ∗ ) = exp ( X i β ) 1 + exp ( X i β ) = 1 1 + exp ( − X i β )
P(Y_i=1) = P(\frac{\epsilon_i}{\sigma_c} \le X_i \beta^*) = \frac{\exp{(X_i \beta)}}{1+\exp{(X_i \beta)}} = \frac{1}{1+\exp{(-X_i\beta)}}
P ( Y i = 1 ) = P ( σ c ϵ i ≤ X i β ∗ ) = 1 + exp ( X i β ) exp ( X i β ) = 1 + exp ( − X i β ) 1 exp ( X i β ) = P ( Y i = 1 ) P ( Y i = 0 ) ⟺ X i β = l n ( P ( Y i = 1 ) P ( Y i = 0 ) ) = l n ( o d d i )
\exp{(X_i\beta)} = \frac{P(Y_i=1)}{P(Y_i=0)} \Longleftrightarrow X_i\beta = ln(\frac{P(Y_i=1)}{P(Y_i=0)}) = ln(odd_i)
exp ( X i β ) = P ( Y i = 0 ) P ( Y i = 1 ) ⟺ X i β = l n ( P ( Y i = 0 ) P ( Y i = 1 ) ) = l n ( o d d i ) P ( Y i = 1 ) P ( Y i = 0 ) \frac{P(Y_i=1)}{P(Y_i=0)} P ( Y i = 0 ) P ( Y i = 1 ) β = ∂ l n ( o d d i ) ∂ X i
\beta = \frac{\partial ln(odd_i)}{\partial X_i}
β = ∂ X i ∂ l n ( o d d i )
定义p i = P ( Y i = 1 ) p_i=P(Y_i=1) p i = P ( Y i = 1 ) Y i Y_i Y i B e r ( p i ) Ber(p_i) B e r ( p i ) f ( Y i ) = p i Y i ( 1 − p i ) 1 − Y i
f(Y_i) = p_i^{Y_i}(1-p_i)^{1-Y_i}
f ( Y i ) = p i Y i ( 1 − p i ) 1 − Y i l ( β ) = ∑ i = 1 N [ Y i l n p i + ( 1 − Y i ) l n ( 1 − p i ) ] = ∑ i = 1 N Y i ln ( p i 1 − p i ) + ∑ i = 1 N l n ( 1 − p i )
l(\beta) = \sum_{i=1}^N [Y_ilnp_i+(1-Y_i)ln(1-p_i)] \\= \sum_{i=1}^N Y_i \ln{(\frac{p_i}{1-p_i})} + \sum_{i=1}^N ln(1-p_i)
l ( β ) = i = 1 ∑ N [ Y i l n p i + ( 1 − Y i ) l n ( 1 − p i ) ] = i = 1 ∑ N Y i ln ( 1 − p i p i ) + i = 1 ∑ N l n ( 1 − p i ) p i p_i p i β \beta β p i = 1 1 + exp ( − X i β ) , 1 − p i = 1 1 + exp ( X i β )
p_i = \frac{1}{1+\exp{(-X_i\beta)}},1-p_i=\frac{1}{1+\exp{(X_i\beta)}}
p i = 1 + exp ( − X i β ) 1 , 1 − p i = 1 + exp ( X i β ) 1 ln ( p i 1 − p i ) = X i β
\ln{(\frac{p_i}{1-p_i})} =X_i\beta
ln ( 1 − p i p i ) = X i β l ( β ) = ∑ i = 1 N Y i X i β − ∑ i = 1 N l n ( 1 + exp ( X i β ) )
l(\beta) = \sum_{i=1}^N Y_i X_i \beta- \sum_{i=1}^N ln(1+\exp{(X_i\beta)})
l ( β ) = i = 1 ∑ N Y i X i β − i = 1 ∑ N l n ( 1 + exp ( X i β ) )
定义对数似然函数的Hessian Matrix为G = ∂ 2 l ( β ) ∂ β 2
G = \frac{\partial^2 l(\beta) }{\partial \beta^2}
G = ∂ β 2 ∂ 2 l ( β ) s 2 ( β ^ ) = ( − G ( β ^ ) ) − 1
s^2(\hat{\beta}) = (-G(\hat{\beta}))^{-1}
s 2 ( β ^ ) = ( − G ( β ^ ) ) − 1
在Logistics回归中，对单个系数的检验需要Wald检验。H 0 : β 1 = 0 H a : β 1 ≠ 0
H_0: \beta_1 = 0 \\
H_a: \beta_1 \ne 0
H 0 : β 1 = 0 H a : β 1 = 0 Z = β ^ 1 − β 1 s ( β ^ 1 ) ∼ N ( 0 , 1 )
Z=\frac{\hat{\beta}_1 - \beta_1}{s(\hat{\beta}_1)} \sim N(0,1)
Z = s ( β ^ 1 ) β ^ 1 − β 1 ∼ N ( 0 , 1 )
似然比检验可以用来对部分或者所有系数进行检验。H 0 : β q = β q + 1 = . . . = β p − 1 = 0 H a : n o t a l l c o e f f i c i e n t s i n H 0 e q u a l t o z e r o
H_0: \beta_q = \beta_{q+1} = ... =\beta_{p-1} = 0 \\
H_a: not\ all\ coefficients\ in\ H_0\ equal\ to\ zero
H 0 : β q = β q + 1 = . . . = β p − 1 = 0 H a : n o t a l l c o e f f i c i e n t s i n H 0 e q u a l t o z e r o L ( R ) L(R) L ( R ) L ( F ) L(F) L ( F ) G 2 = − l n ( L ( R ) L ( F ) ) ∼ χ 2 ( p − q )
G^2 = -ln(\frac{L(R)}{L(F)}) \sim \chi^2(p-q)
G 2 = − l n ( L ( F ) L ( R ) ) ∼ χ 2 ( p − q )
假设对解释变量的取值X j , j = 1 , 2 , . . . , c X_j, j= 1,2,...,c X j , j = 1 , 2 , . . . , c n j n_j n j Y i j , j = 1 , 2 , . . . , n j Y_{ij},j=1,2,...,n_j Y i j , j = 1 , 2 , . . . , n j X j X_j X j Y . j = ∑ i = 1 n j Y i j ∼ B i n o m ( n j , p j )
Y_{.j} = \sum_{i=1}^{n_j} Y_{ij} \sim Binom(n_j,p_j)
Y . j = i = 1 ∑ n j Y i j ∼ B i n o m ( n j , p j ) p j = 1 1 + exp ( − X j β )
p_j = \frac{1}{1+\exp{(-X_j\beta)}}
p j = 1 + exp ( − X j β ) 1
拟合优度检验（Goodness of fit test）的作用是检验模型整体的拟合质量，对小部分拟合不好的地方不会很敏感。如果样本数据存在replication，可以使用卡方拟合优度检验或者Deviance拟合优度检验，如果样本数据不存在replication，可以使用Hosmer-Lemeshow拟合优度检验。Logistics回归的拟合优度检验假设为H 0 : P ( Y i = 1 ) = 1 1 + exp ( − X i β ) H a : P ( Y i = 1 ) ≠ 1 1 + exp ( − X i β )
H_0: P(Y_i=1) = \frac{1}{1+\exp{(-X_i\beta)}} \\
H_a: P(Y_i=1) \ne \frac{1}{1+\exp{(-X_i\beta)}}
H 0 : P ( Y i = 1 ) = 1 + exp ( − X i β ) 1 H a : P ( Y i = 1 ) = 1 + exp ( − X i β ) 1
使用二项回归关于样本replication的设定，O 1 j = Y . j , O 0 j = n j − Y . j
O_{1j} = Y_{.j}, O_{0j} = n_j - Y_{.j}
O 1 j = Y . j , O 0 j = n j − Y . j p ^ j = 1 1 + exp ( − X i β ^ )
\hat{p}_j = \frac{1}{1+\exp{(-X_i\hat{\beta})}}
p ^ j = 1 + exp ( − X i β ^ ) 1 E 1 j = n j p ^ j , E 0 j = n j ( 1 − p ^ j )
E_{1j} = n_j \hat{p}_j, E_{0j} = n_j (1 - \hat{p}_j)
E 1 j = n j p ^ j , E 0 j = n j ( 1 − p ^ j ) p < c p<c p < c n j n_j n j χ 2 = ∑ j = 1 c [ ( O 0 j − E 0 j ) 2 E 0 j + ( O 1 j − E 1 j ) 2 E 1 j ] ∼ χ 2 ( c − p )
\chi^2 = \sum_{j=1}^c [ \frac{(O_{0j}-E_{0j})^2}{E_{0j}} + \frac{(O_{1j}-E_{1j})^2}{E_{1j}} ] \sim \chi^2(c-p)
χ 2 = j = 1 ∑ c [ E 0 j ( O 0 j − E 0 j ) 2 + E 1 j ( O 1 j − E 1 j ) 2 ] ∼ χ 2 ( c − p )
Deviance拟合优度检验用的是似然比检验的思想。其reduced model是Logistics回归，full model是E ( Y i j ) = p j
E(Y_{ij}) = p_j
E ( Y i j ) = p j D e v ( X ) Dev(X) D e v ( X )
Hosmer-Lemeshow拟合优度检验是卡方拟合优度检验的直接推广，n j = 1 n_j=1 n j = 1 n j > 1 n_j>1 n j > 1
假设Y i Y_i Y i m m m 1 , 2 , . . . , m 1,2,...,m 1 , 2 , . . . , m Y i Y_i Y i P ( Y i = j ) = 1 1 + ∑ k = 1 m − 1 exp ( − X i β k ) , j = 1 P ( Y i = j ) = exp ( X i β j ) 1 + ∑ k = 1 m − 1 exp ( − X i β k ) , j > 1
P(Y_i=j) = \frac{1}{1+\sum_{k=1}^{m-1}\exp{(-X_i \beta_k)}}, j = 1 \\ P(Y_i=j) = \frac{\exp{(X_i \beta_j)}}{1+\sum_{k=1}^{m-1}\exp{(-X_i \beta_k)}}, j > 1
P ( Y i = j ) = 1 + ∑ k = 1 m − 1 exp ( − X i β k ) 1 , j = 1 P ( Y i = j ) = 1 + ∑ k = 1 m − 1 exp ( − X i β k ) exp ( X i β j ) , j > 1 j j j m m m j j j p j = e x p ( − ϵ j / k T ) ∑ k = 1 m e x p ( − ϵ k / k T ) = e x p ( ( ϵ 1 − ϵ j ) / k T ) ∑ k = 1 m e x p ( ( ϵ 1 − ϵ k ) / k T )
p_j = \frac{exp(-\epsilon_j/kT)}{\sum_{k=1}^m exp(-\epsilon_k/kT)} = \frac{exp((\epsilon_1-\epsilon_j)/kT)}{\sum_{k=1}^m exp((\epsilon_1-\epsilon_k)/kT)}
p j = ∑ k = 1 m e x p ( − ϵ k / k T ) e x p ( − ϵ j / k T ) = ∑ k = 1 m e x p ( ( ϵ 1 − ϵ k ) / k T ) e x p ( ( ϵ 1 − ϵ j ) / k T ) β k \beta_k β k ϵ k − ϵ 1 \epsilon_k-\epsilon_1 ϵ k − ϵ 1 k k k β k \beta_k β k Y i = k Y_i=k Y i = k 1 1 + ∑ k = 2 m exp ( ϵ 1 − ϵ k ) / k T ) ,
\frac{1}{1+\sum_{k=2}^{m}\exp{(\epsilon_1-\epsilon_k)/kT)}},
1 + ∑ k = 2 m exp ( ϵ 1 − ϵ k ) / k T ) 1 , Y i = 1 Y_i=1 Y i = 1
