Task3:数据重构
通过导入kaggle上泰坦尼克的数据集,实战数据分析全流程
项目地址: 动手学数据分析
开始之前,导入numpy、pandas包和数据
复习:在前面我们已经学习了Pandas基础,第二章我们开始进入数据分析的业务部分,在第二章第一节的内容中,我们学习了数据的清洗,这一部分十分重要,只有数据变得相对干净,我们之后对数据的分析才可以更有力。而这一节,我们要做的是数据重构,数据重构依旧属于数据理解(准备)的范围。
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
import pandas as pd
import numpy as np
df_left_up = pd.read_csv("data/train-left-up.csv")
df_left_up.shape
df_left_up.head()
(439, 4)
|
PassengerId |
Survived |
Pclass |
Name |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
2 第二章:数据重构
2.4 数据的合并
2.4.1 任务一:将data文件夹里面的所有数据都载入,观察数据的之间的关系
df_left_down = pd.read_csv("data/train-left-down.csv")
df_right_up = pd.read_csv("data/train-right-up.csv")
df_right_down = pd.read_csv("data/train-right-down.csv")
df_left_down.shape
df_left_down.head()
(452, 4)
|
PassengerId |
Survived |
Pclass |
Name |
0 |
440 |
0 |
2 |
Kvillner, Mr. Johan Henrik Johannesson |
1 |
441 |
1 |
2 |
Hart, Mrs. Benjamin (Esther Ada Bloomfield) |
2 |
442 |
0 |
3 |
Hampe, Mr. Leon |
3 |
443 |
0 |
3 |
Petterson, Mr. Johan Emil |
4 |
444 |
1 |
2 |
Reynaldo, Ms. Encarnacion |
df_right_up.shape
df_right_up.head()
(439, 8)
|
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
df_right_down.shape
df_right_down.head()
(452, 8)
|
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
male |
31.0 |
0 |
0 |
C.A. 18723 |
10.500 |
NaN |
S |
1 |
female |
45.0 |
1 |
1 |
F.C.C. 13529 |
26.250 |
NaN |
S |
2 |
male |
20.0 |
0 |
0 |
345769 |
9.500 |
NaN |
S |
3 |
male |
25.0 |
1 |
0 |
347076 |
7.775 |
NaN |
S |
4 |
female |
28.0 |
0 |
0 |
230434 |
13.000 |
NaN |
S |
df_train = pd.read_csv("train.csv")
df_train.shape
df_train.head()
(891, 12)
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
【提示】结合之前我们加载的train.csv数据,大致预测一下上面的数据是什么
- 将train.csv数据集划分为左上、左下、右上、右下四个部分
2.4.2:任务二:使用concat方法:将数据train-left-up.csv和train-right-up.csv横向合并为一张表,并保存这张表为result_up
df_result_up = pd.concat([df_left_up, df_right_up], axis=1)
df_result_up.head()
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
df_result_up.to_csv("result_up.csv")
2.4.3 任务三:使用concat方法:将train-left-down和train-right-down横向合并为一张表,并保存这张表为result_down。然后将上边的result_up和result_down纵向合并为result。
df_result_down = pd.concat([df_left_down, df_right_down], axis = 1)
df_result_down.head()
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
440 |
0 |
2 |
Kvillner, Mr. Johan Henrik Johannesson |
male |
31.0 |
0 |
0 |
C.A. 18723 |
10.500 |
NaN |
S |
1 |
441 |
1 |
2 |
Hart, Mrs. Benjamin (Esther Ada Bloomfield) |
female |
45.0 |
1 |
1 |
F.C.C. 13529 |
26.250 |
NaN |
S |
2 |
442 |
0 |
3 |
Hampe, Mr. Leon |
male |
20.0 |
0 |
0 |
345769 |
9.500 |
NaN |
S |
3 |
443 |
0 |
3 |
Petterson, Mr. Johan Emil |
male |
25.0 |
1 |
0 |
347076 |
7.775 |
NaN |
S |
4 |
444 |
1 |
2 |
Reynaldo, Ms. Encarnacion |
female |
28.0 |
0 |
0 |
230434 |
13.000 |
NaN |
S |
df_result_down.to_csv("result_down.csv")
df_result = pd.concat([df_result_up, df_result_down])
df_result.shape
df_result.reset_index(inplace=True,drop=True)
df_train.head()
df_result.head()
(891, 12)
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
```python df_result.equals(df_train) df_result.index ```
True
RangeIndex(start=0, stop=891, step=1)
from pandas.testing import assert_frame_equal
assert_frame_equal(df_result, df_train)
2.4.4 任务四:使用DataFrame自带的方法join方法和append:完成任务二和任务三的任务
df_join_up = df_left_up.join(df_right_up)
df_join_up.shape
df_join_up.head()
(439, 12)
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
df_join_down = df_left_down.join(df_right_down)
df_join_down.shape
df_join_down.head()
(452, 12)
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
440 |
0 |
2 |
Kvillner, Mr. Johan Henrik Johannesson |
male |
31.0 |
0 |
0 |
C.A. 18723 |
10.500 |
NaN |
S |
1 |
441 |
1 |
2 |
Hart, Mrs. Benjamin (Esther Ada Bloomfield) |
female |
45.0 |
1 |
1 |
F.C.C. 13529 |
26.250 |
NaN |
S |
2 |
442 |
0 |
3 |
Hampe, Mr. Leon |
male |
20.0 |
0 |
0 |
345769 |
9.500 |
NaN |
S |
3 |
443 |
0 |
3 |
Petterson, Mr. Johan Emil |
male |
25.0 |
1 |
0 |
347076 |
7.775 |
NaN |
S |
4 |
444 |
1 |
2 |
Reynaldo, Ms. Encarnacion |
female |
28.0 |
0 |
0 |
230434 |
13.000 |
NaN |
S |
df_join = df_join_up.append(df_join_down)
df_join.shape
df_join.head()
df_join.reset_index(inplace=True,drop=True)
df_join.shape
df_join.head()
(891, 12)
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
(891, 12)
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
df_join.equals(df_result)
True
2.4.5 任务五:使用Panads的merge方法和DataFrame的append方法:完成任务二和任务三的任务
df_merge_up = pd.merge(df_left_up, df_right_up, on=[df_left_up.index]).drop("key_0",axis=1)
df_merge_up.shape
df_merge_up.head()
(439, 12)
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
df_merge_up = pd.merge(df_left_up, df_right_up, left_index=True, right_index=True)
df_merge_up.shape
df_merge_up.head()
(439, 12)
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
df_merge_down = pd.merge(df_left_down, df_right_down, left_index=True, right_index=True)
df_merge_down.shape
df_merge_down.head()
(452, 12)
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
440 |
0 |
2 |
Kvillner, Mr. Johan Henrik Johannesson |
male |
31.0 |
0 |
0 |
C.A. 18723 |
10.500 |
NaN |
S |
1 |
441 |
1 |
2 |
Hart, Mrs. Benjamin (Esther Ada Bloomfield) |
female |
45.0 |
1 |
1 |
F.C.C. 13529 |
26.250 |
NaN |
S |
2 |
442 |
0 |
3 |
Hampe, Mr. Leon |
male |
20.0 |
0 |
0 |
345769 |
9.500 |
NaN |
S |
3 |
443 |
0 |
3 |
Petterson, Mr. Johan Emil |
male |
25.0 |
1 |
0 |
347076 |
7.775 |
NaN |
S |
4 |
444 |
1 |
2 |
Reynaldo, Ms. Encarnacion |
female |
28.0 |
0 |
0 |
230434 |
13.000 |
NaN |
S |
df_merge = df_merge_up.append(df_merge_down)
df_merge.shape
df_merge.index
df_merge.reset_index(inplace=True, drop=True)
df_merge.index
df_merge.head()
(891, 12)
Int64Index([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
...
442, 443, 444, 445, 446, 447, 448, 449, 450, 451],
dtype='int64', length=891)
RangeIndex(start=0, stop=891, step=1)
|
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
df_merge.equals(df_train)
True
【思考】对比merge、join以及concat的方法的不同以及相同。思考一下在任务四和任务五的情况下,为什么都要求使用DataFrame的append方法,如何只要求使用merge或者join可不可以完成任务四和任务五呢?
不可以完成任务四和任务五 merge和join的作用是横向链接pandas对象
使用append方法是纵向连接pandas对象
2.4.6 任务六:完成的数据保存为result.csv
df_merge.to_csv("result.csv")
2.5 换一种角度看数据
2.5.1 任务一:将我们的数据变为Series类型的数据
df = pd.read_csv("result.csv")
df.head()
unit_result = df.stack().head()
unit_result.head()
|
Unnamed: 0 |
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
0 Unnamed: 0 0
PassengerId 1
Survived 0
Pclass 3
Name Braund, Mr. Owen Harris
dtype: object
unit_result.to_csv('unit_result.csv')
test = pd.read_csv('unit_result.csv')
test.head()
|
Unnamed: 0 |
Unnamed: 1 |
0 |
0 |
0 |
Unnamed: 0 |
0 |
1 |
0 |
PassengerId |
1 |
2 |
0 |
Survived |
0 |
3 |
0 |
Pclass |
3 |
4 |
0 |
Name |
Braund, Mr. Owen Harris |
上面关于stack函数的讲解是对于多个数组堆叠的情况,在dataframe中不存在多个数组。可将其近似看作每个index和columns对应组成一个1x1的array,然后对其进行axis=0的堆叠,每个array按顺序排放起来。也就相当于是对于dataframe中每个值进行依次排列,做了一个dataframe到Series的解雇转换
复习:在前面我们已经学习了Pandas基础,第二章我们开始进入数据分析的业务部分,在第二章第一节的内容中,我们学习了数据的清洗,这一部分十分重要,只有数据变得相对干净,我们之后对数据的分析才可以更有力。而这一节,我们要做的是数据重构,数据重构依旧属于数据理解(准备)的范围。
开始之前,导入numpy、pandas包和数据
import pandas as pd
import numpy as np
df_result = pd.read_csv("result.csv")
df_result.info()
df_result.head()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 13 columns):
--- ------ -------------- -----
0 Unnamed: 0 891 non-null int64
1 PassengerId 891 non-null int64
2 Survived 891 non-null int64
3 Pclass 891 non-null int64
4 Name 891 non-null object
5 Sex 891 non-null object
6 Age 714 non-null float64
7 SibSp 891 non-null int64
8 Parch 891 non-null int64
9 Ticket 891 non-null object
10 Fare 891 non-null float64
11 Cabin 204 non-null object
12 Embarked 889 non-null object
dtypes: float64(2), int64(6), object(5)
memory usage: 90.6+ KB
|
Unnamed: 0 |
PassengerId |
Survived |
Pclass |
Name |
Sex |
Age |
SibSp |
Parch |
Ticket |
Fare |
Cabin |
Embarked |
0 |
0 |
1 |
0 |
3 |
Braund, Mr. Owen Harris |
male |
22.0 |
1 |
0 |
A/5 21171 |
7.2500 |
NaN |
S |
1 |
1 |
2 |
1 |
1 |
Cumings, Mrs. John Bradley (Florence Briggs Th... |
female |
38.0 |
1 |
0 |
PC 17599 |
71.2833 |
C85 |
C |
2 |
2 |
3 |
1 |
3 |
Heikkinen, Miss. Laina |
female |
26.0 |
0 |
0 |
STON/O2. 3101282 |
7.9250 |
NaN |
S |
3 |
3 |
4 |
1 |
1 |
Futrelle, Mrs. Jacques Heath (Lily May Peel) |
female |
35.0 |
1 |
0 |
113803 |
53.1000 |
C123 |
S |
4 |
4 |
5 |
0 |
3 |
Allen, Mr. William Henry |
male |
35.0 |
0 |
0 |
373450 |
8.0500 |
NaN |
S |
df_result['Sex'] = df_result['Sex'].replace("female", 0).replace('male', 1)
2 第二章:数据重构
第一部分:数据聚合与运算
2.6 数据运用
2.6.1 任务一:通过教材《Python for Data Analysis》P303、Google or anything来学习了解GroupBy机制
写入心得
将数据根据某个(多个)字段划分为不同的群体(group)进行分析,如电商领域将全国的总销售额根据省份进行划分,分析各省销售额的变化情况,社交领域将用户根据画像(性别、年龄)进行细分,研究用户的使用情况和偏好
任何分组(groupby)操作都涉及原始对象的以下操作之一。它们是
在许多情况下,我们将数据分成多个集合,并在每个子集上应用一些函数。在应用函数中,可以执行以下操作
- 聚合 - 计算汇总统计
- 转换 - 执行一些特定于组的操作
- 过滤 - 在某些情况下丢弃数据
2.4.2:任务二:计算泰坦尼克号男性与女性的平均票价
mean_fare = df_result.groupby(df_result['Sex'])['Fare'].mean()
mean_fare
Sex
0 44.479818
1 25.523893
Name: Fare, dtype: float64
在了解GroupBy机制之后,运用这个机制完成一系列的操作,来达到我们的目的。
下面通过几个任务来熟悉GroupBy机制。
2.4.3:任务三:统计泰坦尼克号中男女的存活人数
Survived_Sex_number = df_result.groupby(df_result['Sex'])['Survived'].sum()
Survived_Sex_number
Sex
0 233
1 109
Name: Survived, dtype: int64
2.4.4:任务四:计算客舱不同等级的存活人数
df_result.groupby(df_result['Pclass'])['Survived'].sum()
Pclass
1 136
2 87
3 119
Name: Survived, dtype: int64
【提示:】表中的存活那一栏,可以发现如果还活着记为1,死亡记为0
【思考】从数据分析的角度,上面的统计结果可以得出那些结论
思考心得
- 女性比男性的平均票价更高 说明女性对船舱环境要求高一点,也说明男性对客舱环境要求不高
- 由于较好的客舱环境女性存活的机会更大
- 客舱的等级高存活的机会大,客舱环境差客人警惕性高
【思考】从任务二到任务四中,这些运算可以通过agg()函数来同时计算。并且可以使用rename函数修改列名。你可以按照提示写出这个过程吗?
df_result.groupby(df_result['Sex']).agg({
'Fare':'mean', 'Survived':'sum'})
|
Fare |
Survived |
Sex |
|
|
female |
44.479818 |
233 |
male |
25.523893 |
109 |
df_result.groupby(df_result['Pclass']).agg({
'Survived':'sum'})
|
Survived |
Pclass |
|
1 |
136 |
2 |
87 |
3 |
119 |
#思考心得
df_result.groupby('Survived').agg({
'Sex': 'mean', 'Pclass': 'sum'}).rename(columns=
{
'Sex': 'mean_sex', 'Pclass': 'count_pclass'})
|
mean_sex |
count_pclass |
Survived |
|
|
0 |
0.852459 |
1390 |
1 |
0.318713 |
667 |
2.4.5:任务五:统计在不同等级的票中的不同年龄的船票花费的平均值
df_result.groupby(['Pclass', 'Age'])['Fare'].mean()
Pclass Age
1 0.92 151.5500
2.00 151.5500
4.00 81.8583
11.00 120.0000
14.00 120.0000
...
3 61.00 6.2375
63.00 9.5875
65.00 7.7500
70.50 7.7500
74.00 7.7750
Name: Fare, Length: 182, dtype: float64
2.4.6:任务六:将任务二和任务三的数据合并,并保存到sex_fare_survived.csv
result = pd.merge(mean_fare, Survived_Sex_number, on='Sex')
result.to_csv('sex_fare_survived.csv')
2.4.7:任务七:得出不同年龄的总的存活人数,然后找出存活人数的最高的年龄,最后计算存活人数最高的存活率(存活人数/总人数)
df_survived = df_result.groupby(df_result['Age'])['Survived'].sum()
df_survived
Age
0.42 1
0.67 1
0.75 2
0.83 2
0.92 1
..
70.00 0
70.50 0
71.00 0
74.00 0
80.00 1
Name: Survived, Length: 88, dtype: int64
df_result.groupby(df_result['Age'])['Survived'].sum().index.max()
80.0
people = df_result['Survived'].sum()
people
342
precetn = df_survived.max()/people
precetn
0.043859649122807015