深度学习和机器学习的线性代数入门-FlyAI

深度学习和机器学习的线性代数入门

深度学习

最后更新 2020-11-04 16:02 阅读 17218

最后更新 2020-11-04 16:02

阅读 17218

深度学习

##FlyAI文献翻译
**英文原文：**[：Basic Linear Algebra for Deep Learning and Machine Learning Python Tutorial](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e)
> **标签** ：深度学习

Author(s): Saniya Parveez, Roberto Iriondo

This tutorial’s code is available on [Github](https://github.com/towardsai/tutorials/tree/master/linear-algebra-for-ml-and-deep-learning "Github") and its full implementation as well on [Google Colab](https://colab.research.google.com/drive/1WhzsrEQ-JatDbaaJ81Wh8aYY9jqGWdsw?usp=sharing "Google Colab").

###本文目录：

1. [Introduction](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#46d0 "Introduction")

3. [ Linear Algebra in Machine Learning and Deep Learning](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#60ee " Linear Algebra in Machine Learning and Deep Learning")

5. [Matrix](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#56b6 "Matrix")

7. [Vector](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#a2b1 "Vector")

9. [Matrix Multiplication](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#f25a "Matrix Multiplication")
11. [Transpose Matrix](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#1986 "Transpose Matrix")

13. [Inverse Matrix](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#2173 "Inverse Matrix")

15. [Orthogonal Matrix](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#c383 "Orthogonal Matrix")

17. [Diagonal Matrix](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#fa6e "Diagonal Matrix")

19. [Transpose Matrix and Inverse Matrix in Normal Equation](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#7788 "Transpose Matrix and Inverse Matrix in Normal Equation")
21. [ Linear Equation](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#555f " Linear Equation")
23. [Vector Norms](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#94b3 "Vector Norms")

25. [L1 norm or Manhattan Norm](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#dda9 "L1 norm or Manhattan Norm")

27. [L2 norm or Euclidean Norm](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#11bc "L2 norm or Euclidean Norm")

29. [Regularization in Machine Learning](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#41da "Regularization in Machine Learning")

31. [Lasso](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#564d "Lasso")

33. [Ridge](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#d6fd "Ridge")

35. [Feature Extraction and Feature Selection](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#c54b "Feature Extraction and Feature Selection")

37. [Covariance Matrix](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#953f "Covariance Matrix")

39. [Eigenvalues and Eigenvectors](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#bcd2 "Eigenvalues and Eigenvectors")

41. [Orthogonality](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#9c2e "Orthogonality")

43. [Orthonormal Set](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#4312 "Orthonormal Set")

45. [Span](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#b757 "Span")

47. [Basis](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#f187 "Basis")

49.[ Principal Component Analysis (PCA)](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#7594 " Principal Component Analysis (PCA)")

51. [Matrix Decomposition or Matrix Factorization](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#740e "Matrix Decomposition or Matrix Factorization")

53. [Conclusion](https://medium.com/towards-artificial-intelligence/basic-linear-algebra-for-deep-learning-and-machine-learning-ml-python-tutorial-444e23db3e9e#3904 "Conclusion")

>1. 引言

>3. 机器学习ML和深度学习NN中的线性代数

>5. 矩阵

>7. 向量

>9. 矩阵乘法

>11. 转置矩阵

>13. 逆矩阵

>15. 正交矩阵

>17. 对角矩阵

>19. 正规方程的转置矩阵和逆矩阵

>21. 线性方程

>23. 向量范数

>25. L1范数/Manhattan范数

>27. L2范数/Euclidean范数

>29. ML中的正则化

>31. Lasso

>33. 岭

>35. 特征选择与抽取

>37. 协方差矩阵

>39. 特征值与特征向量

>41. 正交性

>43. 正交集

>45. 扩张空间

>47. 基

>49. 主成分分析(PCA)

>51. 矩阵分解

>53. 总结

###Introduction

The foundation of machine learning and deep learning systems wholly base upon mathematics principles and concepts. It is imperative to understand the fundamental foundations of mathematical principles. During the baseline and building of the model, many mathematical concepts like the curse of dimensionality, regularization, binary, multi-class, ordinal regression, and others must be artistic in mind.

The basic unit of deep learning, commonly called a neuron, is wholly based on its mathematical concept, and such involves the sum of the multiplied values involving input and weight. Its activation functions like Sigmoid, ReLU, and others, have been built using mathematical theorems.

These are the essential mathematical areas to understand the basic concepts of machine learning and deep learning appropriately:

- Linear Algebra.

- Vector Calculus.

- Matrix Decomposition.

- Probability and Distributions.

- Analytic Geometry.

>###引言

>机器学习和深度学习建立在数学原理和概念之上，因此AI学习者需要了解基本数学原理。在模型构建过程中，我们经常设计各种概念，例如维数灾难、正则化、二进制、多分类、有序回归等。

>神经元是深度学习的基本单位，该结构完全基于数学概念，即输入和权重的乘积和。至于Sigmoid，ReLU等等激活函数也依赖于数学原理。

>正确理解机器学习和深度学习的概念，掌握以下这些数学领域至关重要：

>- 线性代数
- 微积分
- 矩阵分解
- 概率论
- 解析几何

###Linear Algebra in Machine Learning and Deep Learning

Linear algebra plays a requisite role in machine learning due to vectors’ availability and several rules to handle vectors. We mostly tackle classifiers or regressor problems in machine learning, and then error minimization techniques are applied by computing from actual value to predicted value. Consequently, we use linear algebra to handle the before-mentioned sets of computations. Linear algebra handles large amounts of data, or in other words, “linear algebra is the basic mathematics of data.”

These are some of the areas in linear algebra that we use in machine learning (ML) and deep learning:

>###机器学习和深度学习中的线性代数

>在机器学习中，很多情况下需要向量化处理，为此，掌握线性代数的知识至关重要。对于机器学习中典型的分类或回归问题，通过最小化实际值与预测值差异进行处理，该过程就用到线性代数。通过线性代数可以处理大量数据，可以这么说，“线性代数是数据科学的基本数学。”

>在机器学习和深度学习中，我们涉及到线性代数的这些知识：

- Vector and Matrix.

- System of Linear Equations.

- Vector Space.

- Basis.

Also, these are the areas of machine learning (ML) and deep learning, where we apply linear algebra’s methods:

- Derivation of Regression Line.

- Linear Equation to predict the target value.

- Support Vector Machine Classification (SVM).

- Dimensionality Reduction.

- Mean Square Error or Loss function.

- Regularization.

- Covariance Matrix.

- Convolution.

>- 向量与矩阵
- 线性方程组
- 向量空间
- 偏差

>通过线性代数，我们可以实现以下机器学习或深度学习方法：

>- 推导回归方程
- 通过线性方程预测目标值
- 支持向量机SVM
- 降维
- 均方差或损失函数
- 正则化
- 协方差矩阵
- 卷积

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263792337421.png)
>矢量积

###Matrix

A matrix is an essential part of linear algebra. It stores m*n elements of data, and we use it for the computation of the linear equation system or the linear mappings. It is an m*n tuple of real-value elements.

>###矩阵

>矩阵是线性代数的重要概念。一个m*n矩阵包含mn个元素，可用于线性方程组或线性映射的计算，也可将其视为一个由m*n个实值元素组成的元组。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263790525714.png)

>矩阵表示

###Vector

In linear algebra, a vector is an n*1 matrix. It has only one column.

>###向量

>在线性代数中，向量是大小为n*1的矩阵，即只有一列。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263790255747.png)

Figure 4: Vector representation.

>矩阵表示

###Matrix Multiplication

Matrix multiplication is a dot product of rows and columns where the row of the matrix is multiplied and summed up with another matrix column.

>###矩阵乘法

>矩阵乘法是行和列的点积，其中一个矩阵的行与另一个矩阵列相乘并求和。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263793384484.png)

###In Linear Regression

There are multiple features to predict the price of a house. Below is the table of different houses with their features and the target value (price).

>###矩阵乘法在线性回归中的应用

>通过多种特征可以预测房屋价格。下表展示了不同房屋的特征及其价格。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263793268208.png)
>不同房屋的特征及其价格

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263792554904.png)

>特征变量和目标变量

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263770628643.png)

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263795296204.png)

>特征及其系数

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263795296204.png)

>房价预测函数

###Transpose Matrix

For A ∈ R^m*n the matrix B ∈ R^n*m with bij = aij is called transpose of A. It is represented as B = A^T.

>###转置矩阵

>对于矩阵A∈R^m*n，有矩阵B∈R^n*m满足b_ij = a_ij，称为A的转置，即B=A^T。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263793584150.png)
>A的转置

###Inverse Matrix

Consider a square matrix A ∈ R^n*n. Let matrix B∈R^n*n has the property that AB = In = BA, B is called the inverse of A and denoted by A^-1.

>###逆矩阵

>对n阶矩阵A，有矩阵B∈R^n*n满足AB =I_n（单位矩阵）= BA的性质，称B为A的逆，表示为A^-1。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263795293864.png)

Figure 11: Matrix A and B.

>矩阵A和B

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263796498372.png)

Figure 12: Multiplication between matrix A and B.

>A和B相乘

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263766807701.png)

>A、B互为逆矩阵（得到单位矩阵）

###Orthogonal Matrix

A square matrix A∈R^n*n is an orthogonal matrix if and only if its columns are orthonormal (unit length) so that:

###正交矩阵

>当且仅当矩阵列向量组是单位正交向量组时，n阶矩阵A∈R^n*n是正交矩阵，有：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263769387478.png)

>正交矩阵

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263793807675.png)

>矩阵A及其转置

###Diagonal Matrix

A square matrix A∈R^n*n is a diagonal matrix where all the elements are zero except those on the main diagonal like:

>Aij =0 for all i != j
>Aij = 0 for some or all i = j

Example:

>###对角矩阵

>在n阶矩阵A∈R^n*n中，除主对角线上的元素，其他所有元素均为零，称其为对角矩阵，即：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263772651402.png)

>对角矩阵

###Transpose Matrix and Inverse Matrix in Normal Equation

The normal equation method minimizes J by explicitly taking its derivatives concerning theta j and setting them to zero. We can directly find out the value of θ without using Gradient Descent [4].

>###正规方程的转置矩阵和逆矩阵

>正规方程通过计算theta j的导数，将其设为零来最小化J。无需Gradient Descent就可直接得到θ的值，θ见下图。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263764101499.png)

>最小化误差

Implementation by taking the data from the table above, “Table1” is shown in figure 5.

>通过上式实现前文“房价预测”。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263794692096.png)

Figure 19: Features under matrix x and prices under matrix y.

Create matrices of features x and target y:

>以矩阵形式表示特征x和目标值y

>创建特征x和目标y的矩阵：

```
import numpy as np Features
x = np.array([[2, 1834, 1],[3, 1534, 2],[2, 962, 3]])# Target or Price
y = [8500, 9600, 258800]
```
Transpose of matrix x:

>计算x的转置：

```
# Transpose of x
transpose_x = x.transpose()transpose_x
```

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263778438475.png)

Figure 20: Transpose of features x.

Multiplication of transposed matrix with original matrix x:

>特征x矩阵的转置

>转置矩阵与原矩阵x的乘积：

```
multi_transpose_x_to_x = np.dot(transpose_x, x)multi_transpose_x_to_x
```

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263771463021.png)

Figure 21: Multiplication of the transposed matrix with the original matrix.

The inverse of the multiplication of the transposed matrix with the original matrix

>转置矩阵与原矩阵x的乘积

>转置矩阵与原始矩阵乘积的逆：

```
inverse_of_multi_transpose_x_to_x = 
np.linalg.inv(multi_transpose_x_to_x)inverse_of_multi_transpose_x_to_x
```
![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263772543672.png)

Figure 22: Inverse matrix.

Multiplication of transposed x with y:

>逆矩阵

>x的转置与y的乘积：

```
multiplication_transposed_x_y = np.dot(transpose_x, 
y)multiplication_transposed_x_y
```
![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263770295736.png)

Figure 23: Multiplication of transposed x with y.

Calculation of theta’s value:

>x的转置与y的乘积

>theta值计算：

```
theta = np.dot(inverse_of_multi_transpose_x_to_x, multiplication_transposed_x_y)theta
```
![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263785637967.png)

>Theta

###Linear Equation

The linear equation is the central part of linear algebra by which many problems are formulated and solved. It is an equation for a straight line.

We represent the linear equation in figure 24:

>###线性方程

>线性方程是线性代数的核心，通过它可以解决许多问题，下图是一条直线方程。

>我们用图24表示线性方程。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263769818471.png)

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263794856459.png)

>线性方程y=4x-5及其图示

>当x=2时：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263771283031.png)

>由上述线性方程式得出的y

###Linear Equations in Linear Regression

Regression is a process that gives the equation for the straight line. It tries to find a best-fitting line with a specific set of data. The equation of the straight line bases on the linear equation:

>Y = bX + a

Where,

a = It is a Y-intercept and determines the point where the line crosses the Y-axis.

b = It is a slope and determines the direction and degree to which the line is tilted.

Implementation

Predict the price of the house where the variables are square feet and price.

Reading the housing price data:

>###线性回归中的线性方程

>回归就是给出线性方程的过程，该过程试图找到满足特定数据集的最优曲线，即：

>Y = bX + a

>其中，a是Y轴截距，决定直线与Y轴相交的点；b是斜率，决定直线倾斜的方向和程度。

>**示例**

>通过线性回归预测平方英尺和房屋价格的关系。

>数据读取：

```
import pandas as pddf = pd.read_csv('house_price.csv')df.head()
```
![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263773593199.png)

>房价表

Calculating the mean:
>计算均值

```
def get_mean(value):
    total = sum(value)
    length = len(value)
    mean = total/length
    return mean
	```

Calculating the variance:

>计算方差：

```
def get_variance(value):
    mean = get_mean(value)
    mean_difference_square = [pow((item - mean), 2) for item in value]
    variance = sum(mean_difference_square)/float(len(value)-1)
    return variance
	```

Calculating the covariance:
>计算协方差：

```
def get_covariance(value1, value2):
    value1_mean = get_mean(value1)
    value2_mean = get_mean(value2)
    values_size = len(value1)
    covariance = 0.0
    for i in range(0, values_size):
        covariance += (value1[i] - value1_mean) * (value2[i] - value2_mean)
    return covariance / float(values_size - 1)
	```

Linear regression implementation:

>线性回归过程：

```
def linear_regression(df):
    X = df['square_feet']
    Y = df['price']
    m = len(X)    square_feet_mean = get_mean(X)
    price_mean = get_mean(Y)
    
    #variance of X
    square_feet_variance = get_variance(X)
    price_variance = get_variance(Y)
    
    covariance_of_price_and_square_feet = get_covariance(X, Y)
    w1 = covariance_of_price_and_square_feet        float(square_feet_variance)    w0 = price_mean - w1 * square_feet_mean
    
    # prediction --> Linear Equation
    prediction = w0 + w1 * X
    
    df['price (prediction)'] = prediction
    return df['price (prediction)']
	```

Calling the ‘linear_regression’ method:

>以上级线性回归方法：

```
linear_regression(df)
```

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263771978701.png)

Figure 29: Predicted price.

The linear equation used in the method “linear_regression”:

>预测价格

>线性回归中的线性方程：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263795435399.png)

###Vector Norms

Vector norms measure the magnitude of a vector [5]. Fundamentally, the size of a given variable x can be represented by its norm ||x||, and the norm represents the distance between two variables x and y, and it is represented by ||x-y||.

The general equation of the vector norm:

>###向量范数

>向量范数可用于衡量向量的大小，也就是说，范数|| x ||表示变量x的大小，范数|| x-y ||表示两个向量x和y之间的距离。

>向量范数计算公式:

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263794804172.png)

Figure 31: Vector norm equation ||x||.

These are the general classes of p-norms:

- L1 norm or Manhattan Norm.

- L2 norm or Euclidean Norm.

L1 and L2 norms are used in Regularization.

>常用的向量范数为一阶和二阶：

>- 一阶范数也叫Manhattan范数
- 二阶范数也叫Euclidean范数

>在正则化中会用到一阶和二阶范数。

###L1 norm or Manhattan Norm

The L1 norm on R^n is defined for x ∈ R^n as shown in figure 31:

>###一阶范数/Manhattan范数

x∈R^n的L1范数定义为：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263778769394.png)

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263774212791.png)

>一阶范式示意图

As shown in figure 32, the red lines symbolize the set of vectors for the L1 norm equation.

>如图32所示，红线象征L1规范方程的向量集。

###L2 norm or Euclidean Norm

The L2 norm of x∈R^n is defined as:

>###L2范数/Euclidean范数

>x∈R^n的L2范数定义为：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263796410818.png)

Figure 34: L2 norm or Euclidean norm equation.

>图34、L2法则或欧几里得法则方程。L2法则或欧氏法则方程。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263786751369.png)

Figure 35: L2 norm representation.

As shown in figure 34, the red lines symbolize the set of vectors for the L2 norm equation.

>图35.L2规范表示。

>如图所示，红线象征着L2规范方程的向量集。

###Regularization in Machine Learning

Regularization is a process of modifying the loss function to penalize specific values of the weight on learning. Regularization helps us avoid overfitting.

It is an excellent addition in machine learning for the operations below:

- To handle collinearity.

- To filter out noise from data.

- To prevent overfitting.

- To get good performance.

These are the standard regularization techniques:

- L1 Regularization (Lasso)

- L2 Regularization (Ridge)

Regularization is the application of the norm.

>###机器学习中的正则化

>正则化是指通过修改损失函数以惩罚学习权重的过程，是避免过拟合的有效方式。

>正则化在机器学习中的作用：

>- 解决共线性问题
- 除去噪声数据
- 避免过拟合
- 提升模型表现

>标准正则化技术包括：

>- L1正则化（Lasso）
- L2正则化（Ridge）

###L1 Regularization (Lasso)

Lasso is a widespread regularization technique. Its formula is shown in figure 35:

>###L1正则化（Lasso）

>Lasso正则化应用广泛，其形式为：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263782951310.png)

###L2 Regularization (Ridge)

The equation of L2 regularization (Ridge):

>###L2正则化（Ridge）

>Ridge正则化表达式：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263797737160.png)

Where, λ = Controls the tradeoff of complexity by adjusting the weight of the penalty term.

>其中，通过λ调整惩罚项的权重进行控制。

###Feature Extraction and Feature Selection

The main intention of feature extraction and feature selection is to pick an optimal set of lower dimensionality features to enhance classification efficiency. These terms essentially treat the curse of the dimensionality problem. Feature selection and feature extraction are performed in the matrix.

>###特征提取和特征选择

>特征提取和特征选择的主要目的是选择一组最佳的低维特征以提高分类效率，以避免维数灾难。在实践中，通过矩阵操作实现特征选择和特征提取。

###Feature Extraction

In feature extraction, we find a set of features from the existing features through some function mapping.

>###特征提取

>在特征提取中，我们通过映射函数从现有特征中找到一组新特征，即:

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263781736980.png)

###Feature Selection

In feature selection, select a subset of the original features.

>###特征选择

>特征选择是指从原始特征中选择部分特征。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263780133874.png)

Main feature extraction methods are:

- Principal Component Analysis (PCA)

- Linear Discriminant Analysis (LDA)

PCA is a critical feature extraction method, and it is vital to know the concepts of the covariance matrix, eigenvalues, or eigenvectors to understand the concept of PCA.

>主要特征抽取方法包括:
- 主成分分析PCA
- 线性判别分析LDA

>其中，PCA是一种典型的特征提取方法，了解协方差矩、特征值或特征向量对于理解PCA至关重要。

###Covariance Matrix

The covariance matrix is an integral part of PCA derivation. The below concepts are important to compute a covariance matrix:

- Variance.

- Covariance.

>###协方差矩阵

>在PCA推导过程中，协方差矩阵起到至关重要的作用。以下两个概念是计算协方差矩阵的基础：

>- 方差
- 协方差

###Variance
>###方差

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263779849400.png)

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263788699362.png)

The limitation of variance is that it does not explore the relationship between variables.

>方差的局限性在于，无法表示变量之间的关系。

###Covariance

Covariance is used to measure the joint variability of two random variables.

>###协方差

>协方差用于衡量两个变量之间的关系：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263797787628.png)

###Covariance Matrix

A covariance matrix is a squared matrix that gives the covariance between each pair of given random vector elements.

>###协方差矩阵

>协方差矩阵是方阵，其中每个元素表示两个随机矢量之间的协方差。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263795830020.png)

The equation of the covariance matrix:

>协方差矩阵的计算公式：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263797382574.png)

###Eigenvalues and Eigenvectors

The definition of Eigenvalues:

> Let m be an n*n matrix. A scalar λ is called the eigenvalue of m if there exists a non-zero vector x in R^n such that mx = λx.

and for Eigenvector:

>The vector x is called an eigenvector corresponding to λ.

###特征值与特征向量

>特征值：

>令m为n*n矩阵，如果存在非零向量x∈R^n，使得mx =λx，则标量λ为矩阵m的特征值。

>特征向量:

>上式中向量x称为特征值λ的特征向量。

###Calculation of Eigenvalues & Eigenvectors

Let m be n*n matrix with eigenvalues λ and corresponding eigenvector x. So, mx = λx. This equation can be written as below:

>###特征值与特征向量的计算

>若n阶矩阵m有特征值λ和相应特征向量x，有mx =λx，则mx —λx= 0，得到下式：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263775127387.png)

Figure 45: Solving this equation for λ gives us all the eigenvalues of m.

Example:

Calculate eigenvalues and eigenvectors of given matrix m:

>求解方程的λ可得到m的所有特征值

>示例：

>计算一下矩阵的特征值和特征向量。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263782268278.png)

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263798203746.png)

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263797120464.png)

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263799611625.png)

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263780615429.png)

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263776919423.png)

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263791541565.png)

Here, the Eigenvalues of m are 2 and -1.

There are multiple eigenvectors available to each eigenvalue.

Orthogonality
Two vectors v and w are called orthogonal if their dot product is zero [7].

v.w = 0

Example:

>因此，矩阵m有两个特征值2和-1。每个特征值对应多个特征向量。

>正交性

>如果向量v和w的点积为零，称两向量正交。

>v.w = 0

>例如：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263798785189.png)

Orthonormal Set

If all vectors in the set are mutually orthogonal and all of the unit lengths, then it is called an Orthonormal set [8]. An orthonormal set that forms a basis is called an orthonormal basis.

Span

Let V is a vector space and its elements are v1, v2, ….., vn ∈ V.

Any sum of these elements multiplied by the scalars representing the equation shown in figure 53, a set of all linear combinations is called the span.

>正交集
如果某一集合中的所有向量相互正交，且均为单位长度，称为规范正交集合。其张成的子空间称为规范正交集。

>扩张空间
令V为向量空间，元素v1，v2，…..，vn∈V。

>将这些元素与标量相乘加和，所有的线性组合集称为扩张空间。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263785994417.png)

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263799599809.png)

>Span (v1, v2, v3) = av1 + bv2 + cv3

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263799808609.png)

Basis
A basis for a vector space is a sequence of vectors that form a set that is linearly independent and that spans the space [9].

Example:

The vector’s sequence below is a basis:

>基

>向量空间的基是一组向量，通过基向量的线性可以组成向量空间中任意一个元素。

>示例：

>假设向量空间的一组基为：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263799308585.png)

It is linearly independent as given below:

>基向量元素是相互独立的，如：

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263789903955.png)

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263788106370.png)

###Principal Component Analysis (PCA)

PCA endeavors a projection that processes as much knowledge in the data as reasonable. It is a dimensionality reduction technique. It finds the directions of the highest variance and projects the data with them to decrease the dimensions.

Calculation steps of PCA:

Let there is an N*1 vector with values x1, x2, ….., xm.

Calculate the sample mean:

>###主成分分析PCA

>通过PCA能够对数据进行降维，以处理尽可能多的数据。其原理是：找到方差最大的方向，在该方向上进行投影以减小数据维度。

>PCA的计算方法：

>设有一个N*1向量，其值为x1，x2，…..，xm。

>1.计算样本均值

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263783495585.png)

ubtract sample mean with vector value:
>向量元素减均值

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263782360790.png)

Calculate the sample covariance matrix:

>计算样本协方差矩阵

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263787518395.png)

Calculate the eigenvalues and eigenvectors of the covariance matrix

>计算协方差矩阵的特征值和特征向量

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263798620474.png)

Dimensionality reduction: approximate x using only the first k eigenvectors (k< N).

>降维：选择前k个特征向量近似x（k <N）

**Python Implementation of the Principal Component Analysis (PCA)**

The main goal of Python’s implementation of the PCA:

- Implement the covariance matrix.

- Derive eigenvalues and eigenvectors.

- Understand the concept of dimensionality reduction from the PCA.

Loading the Iris data

>**python实现主成分分析**

>为实现PCA，需要实现以下功能：

>- 获取协方差矩阵
- 计算特征值和特征向量
- 通过PCA了解降维

>Iris数据导入

```
import numpy as np
import pylab as pl
import pandas as pd
from sklearn 
import datasets
import matplotlib.pyplot as pltfrom sklearn.preprocessing 
import StandardScalerload_iris = datasets.load_iris()iris_df = pd.DataFrame(load_iris.data, columns=[load_iris.feature_names])iris_df.head()
```

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263799685096.png)

**Standardization**

It is always good to standardize the data to keep all features of the data in the same scale.

>**标准化**

>标准化数据使得所有特征数据处于同一量级区间，有利于分析特征。

```
standardized_x = 
StandardScaler().fit_transform(load_iris.data)standardized_x[:2]
```
![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263784508502.png)

**Compute Covariance Matrix**

>计算协方差矩阵

```
covariance_matrix_x = np.cov(standardized_x.T)covariance_matrix_x
```
![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263787403140.png)

Compute Eigenvalues and Eigenvectors from Covariance Matrix

>计算协方差矩阵得特征值和特征向量

```
eigenvalues, eigenvectors = np.linalg.eig(covariance_matrix_x)

eigenvalues
```

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263787872472.png)
```
eigenvectors
```
![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263765168553.png)

Check variance in Eigenvalues
>特征值的方差

```
total_of_eigenvalues = sum(eigenvalues)varariance = [(i / 
total_of_eigenvalues)*100 for i in sorted(eigenvalues, 
reverse=True)]varariance
```
![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263798853745.png)

The values shown in figure 68 indicate the variance giving the analysis below:

- 1st Component = 72.96%

- 2nd Component = 22.85%

- 3rd Component = 3.5%

- 4th Component = 0.5%

So, the third and fourth Components have very low variance respectively. These can be dropped. Because these components can’t add any value.

Taking 1st and 2nd Components only and Reshaping

>上图中的方差值分别表示：

>- 1st 成分 = 72.96%

>- 2nd   成分  = 22.85%

>- 3rd  成分  = 3.5%

>- 4th  成分  = 0.5%

>可以看到，第三和第四成分具有很小的方差，可以忽略不记，这些组分不会对最终结果产生太大影响。

>保留占比大的第一、第二成分，并进行以下操作：

```
eigenpairs = [(np.abs(eigenvalues[i]), eigenvectors[:,i]) for i in range(len(eigenvalues))]
# Sorting from Higher values to lower valueeigenpairs.sort(key=lambda x: x[0], reverse=True)eigenpairs
```
![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263765931628.png)

Multiply the standardized matrix with matrix weighing:

>将标准化矩阵乘以矩阵权重

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263769600284.png)

>绘图

```
plt.figure()target_names = load_iris.target_names
y = load_iris.targetfor c, i, target_name in zip("rgb", [0, 1, 2], target_names):
    plt.scatter(Y[y==i,0], Y[y==i,1], c=c, label=target_name)plt.xlabel('PCA 1')
plt.ylabel('PCA 2')
plt.legend()
plt.title('PCA')
plt.show()
```
![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263777235579.png)

The plotted result of the PCA 1 and PCA 2 from the first dataset.

>Iris数据的主成分分析示意图

**Matrix Decomposition or Matrix Factorization**

Matrix Decomposition or factorization is also an important part of linear algebra used in machine learning. Basically, it is a factorization of the matrix into a product of matrices.

There are several techniques of matrix decomposition like LU decomposition, Singular value decomposition (SVD), etc.

Singular Value Decomposition (SVD)

It is a technique for the reduction of dimension. As per singular value decomposition matrix:

Let M is a rectangular matrix and can be broken down into three products of matrix — (1) orthogonal matrix (U), (2) diagonal matrix (S), and (3) transpose of the orthogonal matrix (V).

>**矩阵分解**

>矩阵分解在机器学习中也至关重要，该过程本质上是将矩阵分解为矩阵的乘积。

>常用的矩阵分解技术有LU分解，奇异值分解（SVD）等。

>奇异值分解（SVD）

>SVD可用于减小数据维度，奇异值分解如下：

>令M为矩阵，其可以分解为三个矩阵的乘积，即正交矩阵（U），对角矩阵（S）和正交矩阵（V）的转置。

![](https://static.leiphone.com/uploads/new/sns/article/202010/1603263768537775.png)

**Conclusion**

Machine learning and deep learning has been build upon the concept of mathematics. A vast area of mathematics is used to build algorithms and also for the computation of data.

Linear algebra is the study of vectors [10] and several rules to manipulate vectors. It is a key infrastructure, and it covers many areas of machine learning like linear regression, one-hot encoding in the categorical variable, PCA (Principle component analysis) for the dimensionality reduction, matrix factorization for recommender systems.

Deep learning is completely based on linear algebra and calculus. It is also used in several optimization techniques like gradient descent, stochastic gradient descent, and others.

Matrices are an essential part of linear algebra that we use to compactly represent systems of linear equations, linear mapping, and others. Also, vectors are unique objects that can be added together and multiplied by scalars that produce another object of similar kinds. Any suggestions or feedback is crucial to continue to improve. Please let us know in the comments if you have any.

DISCLAIMER: The views expressed in this article are those of the author(s) and do not represent the views of Carnegie Mellon University, nor other companies (directly or indirectly) associated with the author(s). These writings do not intend to be final products, yet rather a reflection of current thinking, along with being a catalyst for discussion and improvement.

Published via Towards AI

>**结论**

>机器学习和深度学习是建立在数学概念之上的，掌握理解数学知识对于算法构建和数据处理有极大帮助。

>线性代数的研究包括向量及其操作。在机器学习中，各处可见线性代数的背影，如线性回归，独热编码，主成分分析PCA，推荐系统中的矩阵分解。

>深度学习更甚，其完全基于线性代数和微积分。梯度下降，随机梯度下降等优化方法也建立在此之上。

>矩阵是线性代数中的重要概念，通过矩阵可以紧凑地表示线性方程组、线性映射等。同样，向量也是重要的概念，将不同向量与标量乘积进行加和可以组成不同的向量空间。

>欢迎在评论区回复你的看法，我会虚心接受并进行改进。

>免责声明：本文中表达的观点仅为作者个人观点，不（直接或间接）代表卡耐基梅隆大学或与作者相关的其他组织。我知道，本文不尽完善，仅是本人当前的一些看法与思考，希望对读者有所帮助。

**Resources:**

[Google colab implementation.](https://colab.research.google.com/drive/1WhzsrEQ-JatDbaaJ81Wh8aYY9jqGWdsw?usp=sharing "Google colab implementation.")

[Github repository.](https://github.com/towardsai/tutorials/tree/master/linear-algebra-for-ml-and-deep-learning "Github repository.")

**参考**

[1] Linear Algebra, Wikipedia, https://en.wikipedia.org/wiki/Linear_algebra

[2] Euclidean Space, Wikipedia, https://en.wikipedia.org/wiki/Euclidean_space

[3] High-dimensional Simplexes for Supermetric Search, Richard Connor, Lucia Vadicamo, Fausto Rabitti, ResearchGate, https://www.researchgate.net/publication/318720793_High-Dimensional_Simplexes_for_Supermetric_Search

[4] ML | Normal Equation in Linear Regression, GeeksforGeeks, https://www.geeksforgeeks.org/ml-normal-equation-in-linear-regression/

[5] Vector Norms by Roger Crawfis, CSE541 — Department of Computer Science, Stony Brook University, https://www.slideserve.com/jaimie/vector-norms

[6] Variance Estimation Simulation, Online Stat Book by Rice University, http://onlinestatbook.com/2/summarizing_distributions/variance_est.html

[7] Lecture 17: Orthogonality, Oliver Knill, Harvard University, http://people.math.harvard.edu/~knill/teaching/math19b_2011/handouts/math19b_2011.pdf

[8] Orthonormality, Wikipedia, https://en.wikipedia.org/wiki/Orthonormality

[9] Linear Algebra/Basis, Wikibooks, https://en.wikibooks.org/wiki/Linear_Algebra/Basis

[10] Linear Algebra, LibreTexts, https://math.libretexts.org/Bookshelves/Linear_Algebra

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可，转载请附上原文出处链接和本声明。
本文链接地址：https://www.flyai.com/article/746

讨论

500字

表情

发送

删除确认

是否删除该条评论？

取消删除