Manually calculate the mean, variance, covariance, Pearson coefficient

Project github address: bitcarmanlee easy-algorithm-interview-and-practice
welcome everyone to star, leave a message, and learn and progress together

1. Mean, variance

Suppose there is an array with a total of n elements, then the calculation formula for mean and variance is
mean:
$$\bar{x}=\frac{1}{n}\sum _{i=1}^n{x}_i$$

Variance (overall variance):
$$var=\frac{1}{n}\sum _{i=1}^n(x_i-\bar{x}{)}^2$$

Variance (sample variance):
$$S=\frac{1}{n-1}\sum _{i=1}^n(x_i-\bar{x}{)}^2$$

Among them, the overall variance is divided by n, and the sample variance is divided by n-1.

2. Covariance and Pearson's coefficient

Covariance (Covariance) is used to measure the degree of joint variation of two random variables. Therefore, when calculating the covariance, you need to enter two variables. Variance is a special case of covariance, that is, the covariance between a variable and itself.

$$cov(X,And)=\frac{{\sum }_{i=1}^n(x_i-\bar{x})({and}_i-\bar{Y})}{n-1}$$

If the denominator is divided by n-1, then the covariance of the sample is calculated. If it is divided by n, the overall covariance is calculated.

The Pearson correlation coefficient between two variables is the quotient of the covariance and standard deviation between the two variables:
$${\rho }_{x,and}=\frac{cov(X,Y)}{{\sigma }_x{\sigma }_{and}}=\frac{And\left[\right(X-{\mu }_x)(X-{\mu }_{and})]}{{\sigma }_x{\sigma }_{and}}$$

The Pearson coefficient of the sample, often with the letter $r$ means:

$$r=\frac{{\sum }_{i=1}^n(X_i-\overline{X})({And}_i-\overline{Y})}{\sqrt{{\sum }_{i=1}^n(X_i-\overline{X}{)}^2}\sqrt{{\sum }_{i=1}^n({And}_i-\overline{Y}{)}^2}}$$

The Pearson coefficient r is a dimensionless number that measures linear independence, and its value is between [-1, 1]. If r=1, it means a complete linear correlation. If r=-1, it means a completely linear negative correlation. r=0, that is, the cov value is 0, indicating that the two variables are uncorrelated, or more accurately called "linearly independent" or "linearly uncorrelated", indicating that there is no linear correlation between the two random variables X and Y, not It means that there must be no intrinsic (non-linear) functional relationship between them, and X and Y are not necessarily statistically independent.

3. Code implementation

So many theories have been mentioned before, then we will look at the specific implementation.

import numpy as np
import math
import pandas as pd

def t1():
    a = [5, 6, 16, 9]
    print(np.mean(a))
    print()

    # 整体方差除以n
    var = np.var(a)
    print(var)
    # 自己计算方差
    var2 = [math.pow(x-np.mean(a), 2) for x in a]
    print(np.mean(var2))
    print()

    # 样本方差除以n-1
    sample_var = np.var(a, ddof=1)
    print(sample_var)
    print()

    # 标准差
    std = np.std(a)
    std2 = np.std(a, ddof=1)
    print(std)
    print(std2)


t1()

Output result:

9.0

18.5
18.5

24.666666666666668

4.301162633521313
4.96655480858378

Let’s take a look at the calculation of Pearson’s coefficient.

def t2():
    a = [i for i in range(10)]
    b = [2*x + 0.1 for x in a]

    dic = {"x": a, "y": b}
    df = pd.DataFrame(dic)
    print(df)
    print()
    print("dfcorr is: ", df.corr())

    # 手动计算pearson系数
    n = len(a)
    abar = sum(a)/float(len(a))
    bbar = sum(b)/float(len(b))
    covab = sum([(x-abar)*(y-bbar) for (x, y) in zip(a, b)]) / n

    stda = math.sqrt(sum([math.pow(x-abar, 2) for x in a]) / n)
    stdb = math.sqrt(sum([math.pow(y-bbar, 2) for y in b]) / n)

    print()
    print(covab)
    print(stda)
    print(stdb)
    print()
    corrnum = covab / (stda * stdb)
    print("corrnum is: ", corrnum)


t2()

Output result

   x     y
0  0   0.1
1  1   2.1
2  2   4.1
3  3   6.1
4  4   8.1
5  5  10.1
6  6  12.1
7  7  14.1
8  8  16.1
9  9  18.1

dfcorr is:       x    y
x  1.0  1.0
y  1.0  1.0

16.5
2.8722813232690143
5.7445626465380295

corrnum is:  0.9999999999999998

x and y are two variables respectively, y = 2x + 0.1, the two have a complete linear correlation, so the pearson coefficient calculated at the end is 1.0.