LDA线性判别式分析又称为Fisher线性判别，是一种有监督的降维算法，用于针对有类别的样本进行降维，使得降维后类与类之间的分割依然很明显，可以说，LDA从高维特征提取出了最具有类间判别能力低维特征，LDA与PCA分类的区别如图1所示，红色与蓝色代表不同类别的样本。
u u u u u u x j i x_j^i x j i m j m_j m j D j D_j D j x j i ~ \widetilde{x_j^i} x j i x j i x_j^i x j i u u u m j ~ \widetilde{m_j} m j m j m_j m j u u u u u u a a a
为第 j 类的第 i 个样本 。
计算类内距离：
S j = 1 N j ∑ x j i ∈ D j ( x j i ~ − m j ~ ) T ( x j i ~ − m j ~ ) = 1 N j ∑ x j i ∈ D j ( u T x j i u − u T m j u ) T ( u T x j i u − u T m j u ) = a 2 N j ∑ x j i ∈ D j ( u T x j i u T − u T m j u T ) ( u T x j i u − u T m j u ) = a 2 N j ∑ x j i ∈ D j [ ( u T x j i ) 2 − 2 u T x j i u T m j + ( u T m j ) 2 ] = a 2 ( 1 N j ∑ x j i ∈ D j u T x j i x j i T u − 2 N j ( ∑ x j i ∈ D j x j i T ) ⋅ ( u m j T u ) + ( u T m j ) 2 ) = a 2 ( u T N j ( ∑ x j i ∈ D j x j i x j i T ) ⋅ u − 2 ( m j T u ) 2 + ( u T m j ) 2 ) = a 2 ( u T N j ( ∑ x j i ∈ D j x j i x j i T ) ⋅ u − u T m j m j T ⋅ u ) = a 2 u T ( ( 1 N j ∑ x j i ∈ D j x j i x j i T ) − m j m j T ) u
\begin{aligned}
Sj &= \frac{1}{N_j} \sum_{x_j^i \in D_j}(\widetilde{x_j^i} - \widetilde{m_j})^T(\widetilde{x_j^i} - \widetilde{m_j})\\
&= \frac{1}{N_j}\sum_{x_j^i \in D_j}(u^Tx_j^iu - u^Tm_ju)^T(u^Tx_j^iu-u^Tm_ju)\\
&= \frac{a^2}{N_j}\sum_{x_j^i \in D_j}(u^Tx_j^iu^T - u^Tm_ju^T)(u^Tx_j^iu-u^Tm_ju)\\
&= \frac{a^2}{N_j}\sum_{x_j^i \in D_j}[(u^Tx_j^i)^2 - 2u^Tx_j^iu^Tm_j + (u^Tm_j)^2]\\
&= a^2(\frac{1}{N_j}\sum_{x_j^i \in D_j}u^Tx_j^ix_j^{iT}u - \frac{2}{N_j}(\sum_{x_j^i \in D_j}x_j^{iT})\cdot(um_j^Tu) + (u^Tm_j)^2)\\
&= a^2(\frac{u^T}{N_j}(\sum_{x_j^i \in D_j}x_j^ix_j^{iT})\cdot u - 2(m_j^Tu)^2 + (u^Tm_j)^2) \\
&= a^2(\frac{u^T}{N_j}(\sum_{x_j^i \in D_j}x_j^ix_j^{iT})\cdot u - u^T m_jm_j^T\cdot u)\\
&= a^2u^T((\frac{1}{N_j}\sum_{x_j^i \in D_j}x_j^ix_j^{iT}) - m_jm_j^T)u
\end{aligned}
S j = N j 1 x j i ∈ D j ∑ ( x j i − m j ) T ( x j i − m j ) = N j 1 x j i ∈ D j ∑ ( u T x j i u − u T m j u ) T ( u T x j i u − u T m j u ) = N j a 2 x j i ∈ D j ∑ ( u T x j i u T − u T m j u T ) ( u T x j i u − u T m j u ) = N j a 2 x j i ∈ D j ∑ [ ( u T x j i ) 2 − 2 u T x j i u T m j + ( u T m j ) 2 ] = a 2 ( N j 1 x j i ∈ D j ∑ u T x j i x j i T u − N j 2 ( x j i ∈ D j ∑ x j i T ) ⋅ ( u m j T u ) + ( u T m j ) 2 ) = a 2 ( N j u T ( x j i ∈ D j ∑ x j i x j i T ) ⋅ u − 2 ( m j T u ) 2 + ( u T m j ) 2 ) = a 2 ( N j u T ( x j i ∈ D j ∑ x j i x j i T ) ⋅ u − u T m j m j T ⋅ u ) = a 2 u T ( ( N j 1 x j i ∈ D j ∑ x j i x j i T ) − m j m j T ) u S i n = ∑ j = 1 K S j = a 2 u T ∑ j = 1 K ( 1 N j ∑ x j i ∈ D j x j i x j i T ) − m j m j T ) u
\begin{aligned}
S_{in} &= \sum_{j=1}^{K}S_j\\
&= a^2u^T\sum_{j=1}^{K} (\frac{1}{N_j}\sum_{x_j^i\in D_j}x_j^ix_j^{iT}) - m_jm_j^T)u
\end{aligned}
S i n = j = 1 ∑ K S j = a 2 u T j = 1 ∑ K ( N j 1 x j i ∈ D j ∑ x j i x j i T ) − m j m j T ) u u u u s w = ∑ j = 1 K ( 1 N j ∑ x j i ∈ D j x j i x j i T ) − m j m j T ) s_w = \sum_{j=1}^{K} (\frac{1}{N_j}\sum_{x_j^i\in D_j}x_j^ix_j^{iT}) - m_jm_j^T) s w = ∑ j = 1 K ( N j 1 ∑ x j i ∈ D j x j i x j i T ) − m j m j T ) u u u u T s w u u^Ts_wu u T s w u S i j = ( m i ~ − m j ~ ) T ( m i ~ − m j ~ ) = ( u T m i u − u T m j u ) T ( u T m i u − u T m j u ) = u T ( m i − m j ) u T ⋅ u T ( m i − m j ) u = a 2 u T ( m i − m j ) u T ( m i − m j ) = a 2 u T ( m i − m j ) ( m i − m j ) T u
\begin{aligned}
S_{ij} &= (\widetilde{m_i} - \widetilde{m_j})^T(\widetilde{m_i} - \widetilde{m_j})\\
&= (u^Tm_iu - u^Tm_ju)^T(u^Tm_iu - u^Tm_ju)\\
&= u^T(m_i - m_j)u^T\cdot u^T(m_i - m_j)u\\
&= a^2u^T(m_i - m_j)u^T(m_i-m_j)\\
&= a^2u^T(m_i - m_j)(m_i-m_j)^Tu
\end{aligned}
S i j = ( m i − m j ) T ( m i − m j ) = ( u T m i u − u T m j u ) T ( u T m i u − u T m j u ) = u T ( m i − m j ) u T ⋅ u T ( m i − m j ) u = a 2 u T ( m i − m j ) u T ( m i − m j ) = a 2 u T ( m i − m j ) ( m i − m j ) T u S o u t = ∑ i < j i < K , j < K S i j = a 2 u T ( ∑ i < j i < K , j < K ( m i − m j ) ( m i − m j ) T ) u
\begin{aligned}
S_{out} &= \sum_{i<j}^{i<K, j<K}S_{ij}\\
&= a^2u^T(\sum_{i<j}^{i<K, j<K}(m_i - m_j)(m_i-m_j)^T)u
\end{aligned}
S o u t = i < j ∑ i < K , j < K S i j = a 2 u T ( i < j ∑ i < K , j < K ( m i − m j ) ( m i − m j ) T ) u s b = ∑ i < j i < K , j < K ( m i − m j ) ( m i − m j ) T s_b = \sum_{i<j}^{i<K, j<K}(m_i - m_j)(m_i-m_j)^T s b = ∑ i < j i < K , j < K ( m i − m j ) ( m i − m j ) T S o u t S_{out} S o u t u T s b u u^Ts_bu u T s b u
为了同时满足最小化u T s w u u^Ts_wu u T s w u u T s b u u^Ts_bu u T s b u u u u u T s w u = 1 u^Ts_wu=1 u T s w u = 1 u T s b u u^Ts_bu u T s b u f ( u ) = u T s b u + λ ( 1 − u T s w u )
f(u) = u^Ts_bu+\lambda (1-u^Ts_wu)
f ( u ) = u T s b u + λ ( 1 − u T s w u ) ∂ f u = s b u − λ s w u = 0
\frac{\partial f}{u} = s_bu - \lambda s_wu = 0
u ∂ f = s b u − λ s w u = 0 u u u a 2 u T s w u ≥ 0 a^2u^Ts_wu\ge 0 a 2 u T s w u ≥ 0 u T s w u > 0 u^Ts_wu\gt 0 u T s w u > 0 s w s_w s w s w s_w s w s w − 1 s b u = λ u
s_w^{-1}s_bu = \lambda u
s w − 1 s b u = λ u s w s_w s w s b s_b s b u u u s w − 1 s b s_w^{-1}s_b s w − 1 s b
当应用统计方法解决人脸识别问题时，一再碰到维数问题和小样本问题,因此降低维数就成为处理实际问题的关键。线性判别分析LDA (LinearDiscriminant Analysis)， 或者称为Fisher线性判别是其中较为常用的方法。但在应用LDA常常遇到“小样本"问题。研究者们提出了很多方法解决它。Swets和Weng"提出了PCA+LDA的两步策略,利用主成分分析PCA (Prineipal Component Analysis)将 高维空间的样本投影到低维空间以保证样本类内离散度矩阵是非奇异的[1] 。然而该方法的问题在于PCA在降维的同时也丢失了很多帮助判别的有用信息。Yu和Yang提出的DLDA2采用首先删除样本类间离散度矩阵的零空间,然后将样本类内离散度矩阵投影到样本类间离散度矩阵的非零空间。再选择相对投影后的样本类内离散度矩阵的较大特征值的特征向量作为变换矩阵[2] 。但是因为要设置阈值使得具体实现不是很方便。此后,Lu等人提出了VDLDAB.4，加入了加权变量,并改变了最优Fisher准则函数的形式[3, 4] 。随后又加入了Fractional-stepLDAS,不仅简洁有效地解决了维数问题和小样本问题,而且很大程度提高了识别率[5] 。[6] 。
[1] P Kocher, J Jaffe,B Jun. Differential Power Analysis and
