TPOT分析水文数据

1. 水文数据集

https://data.edmonton.ca/dataset/Water-Levels-and-Flows/cnsu-iagr

###该数据集一共9个属性:
1.Raw ID 
2.Station Number => 水文站编号  --26 --离散值
3.Station Description => 水文站描述 --26 --文本信息
4.Latitude => 纬度 --28 --离散值
5.Longitude => 经度 --28 --离散值
6.Station Location => 水文站位置，由经纬度组合而成  --28 --离散值
7.Date and Time => 时间
8.Water Level (m) => 水位 -- --连续值
9.Water Flow (cu meter per sec) => 流速 -- --连续值

因为TPOT只能处理数值类型，在我们的数据集中，有4个变量不是数值类型:
Station Number, Station Description, Station Location, Date and Time
所以，下面重点是处理这几个变量；

###数据预处理:
1.Station Number
离散值，有26个可选，使用one-hot编码

2.Station Description， Station Location, Raw ID
无用信息，删除

3.Date and Time
分析数据，从2016年到2019年，所以年份不敏感；
拆分为月，日和小时

4.Latitude和Longitude可能缺失
总的记录数:
(3910803, 9)

非空记录数:
(1975617, 9)

5.经纬度缺失的数据量很大，所以不能直接舍弃；
首先计算水文站经纬度的平均值，用这个来代替

6.Water Flow (cu meter per sec)可能非法，例如-1, 删除这些记录

2. 代码

from tpot import TPOTRegressor
from sklearn.model_selection import train_test_split
import pandas as pd
import numpy as np
import os
from tensorflow.python.client import device_lib

os.environ["CUDA_VISIBLE_DEVICES"] = "1"
print(device_lib.list_local_devices())

df = pd.read_csv('Water_Levels_and_Flows.csv.1w', header=0,error_bad_lines=False)
print(df.head(5))
print(df.shape)
print(pd.isnull(df).any())

#删除Water_Flow为负的行
print('删除Water_Flow无效行')
df=df[~(df['Water_Flow']<0)]
print(df.shape)

mean_longitude = df.groupby('Station Number').Longitude.mean()
mean_latitude = df.groupby('Station Number').Latitude.mean()
null_longitude = df[df.Longitude.isnull()]

for cat in ['Station Number', 'Station Description', 'Latitude', 'Longitude' ]:
    print("Number of levels in category '{0}': \b {1:2.2f} ".format(cat, df[cat].unique().size))

for cat in ['Latitude', 'Longitude']:
    print("Levels for catgeory '{0}': {1}".format(cat, df[cat].unique()))

#删掉无关因素
df = df.drop(['id', 'Station Description','Station Location', 'Latitude', 'Longitude'],axis=1) # axis=0 删除行，=1 删除列

#将Station用哑变量表示
dumm = pd.get_dummies(df[['Station Number']]) # '哑变量'矩阵
df = df.join(dumm)
del df['Station Number'] # 删除

#添加month,day,hour三个键和值
temp=pd.DatetimeIndex(df['Date and Time'])
df['date']=temp.date
df['time']=temp.time
df['hour']=pd.to_datetime(df.time,format="%H:%M:%S")#变换格式
df['hour']=pd.Index(df["hour"]).hour
df['month']=pd.to_datetime(df.date,format="%Y-%m-%d")#变换格式
df['month']=pd.Index(df["month"]).month
df['day']=pd.to_datetime(df.date,format="%Y-%m-%d")#变换格式
df['day']=pd.Index(df["day"]).day
df = df.drop(['Date and Time', 'date', 'time'],axis=1) # axis=0 删除行，=1 删除列
print(df.head(5))
print(df.shape)

X = np.array(df.drop(['Water_Level'], 1))
Y = np.array(df['Water_Level'])
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2)

tpot = TPOTRegressor(generations=20, verbosity=2) #迭代20次
tpot.fit(X_train, y_train)
print(tpot.score(X_test, y_test))
tpot.export('pipeline.py')

3. 运行TPOT

因为执行时间过长，所以使用后台运行方式:

nohup python -u water_tpot.py > nohup.out 2>&1 &

输出如下:

Generation 1 - Current best internal CV score: -0.10354635942452688
Generation 2 - Current best internal CV score: -0.07407627082459693
Generation 3 - Current best internal CV score: -0.07407627082459693
Generation 4 - Current best internal CV score: -0.07407627082459693
Generation 5 - Current best internal CV score: -0.07261441673202419
Generation 6 - Current best internal CV score: -0.07261441673202419
Generation 7 - Current best internal CV score: -0.07261441673202419
Generation 8 - Current best internal CV score: -0.06272878450716013
Generation 9 - Current best internal CV score: -0.06150379726583473
Generation 10 - Current best internal CV score: -0.06150379726583473
Generation 11 - Current best internal CV score: -0.06150379726583473
Generation 12 - Current best internal CV score: -0.05930206668688394
Generation 13 - Current best internal CV score: -0.054799951152979184
Generation 14 - Current best internal CV score: -0.054799951152979184
Generation 15 - Current best internal CV score: -0.052015877476651276
Generation 16 - Current best internal CV score: -0.052015877476651276
Generation 17 - Current best internal CV score: -0.05173009225925466
Generation 18 - Current best internal CV score: -0.043718068024817355
Generation 19 - Current best internal CV score: -0.043718068024817355
Generation 20 - Current best internal CV score: -0.043718068024817355
Best pipeline: ExtraTreesRegressor(SelectPercentile(DecisionTreeRegressor(ElasticNetCV(input_matrix, l1_ratio=0.7000000000000001, tol=1e-05), max_depth=1, min_samples_leaf=2, min_samples_split=15), percentile=94), bootstrap=False, max_features=1.0, min_samples_leaf=2, min_samples_split=2, n_estimators=100)
-0.027241579610884296

TPOT认为ExtraTreesRegressor是最好的模型，并导出pipeline.py。

4. 运行pipeline

导出的pipeline.py文件需要根据实际情况修改如下:

import numpy as np
import pandas as pd
from sklearn.ensemble import ExtraTreesRegressor
from sklearn.feature_selection import SelectPercentile, f_regression
from sklearn.linear_model import ElasticNetCV
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline, make_union
from sklearn.tree import DecisionTreeRegressor
from tpot.builtins import StackingEstimator

# NOTE: Make sure that the class is labeled 'target' in the data file
df = pd.read_csv('Water_Levels_and_Flows.csv.1w', sep=',', header=0,error_bad_lines=False)
#删除Water_Flow为负的行
print('删除Water_Flow无效行')
df=df[~(df['Water_Flow']<0)]
print(df.shape)

mean_longitude = df.groupby('Station Number').Longitude.mean()
mean_latitude = df.groupby('Station Number').Latitude.mean()
null_longitude = df[df.Longitude.isnull()]

for cat in ['Station Number', 'Station Description', 'Latitude', 'Longitude' ]:
    print("Number of levels in category '{0}': \b {1:2.2f} ".format(cat, df[cat].unique().size))

for cat in ['Latitude', 'Longitude']:
    print("Levels for catgeory '{0}': {1}".format(cat, df[cat].unique()))

#删掉无关因素
df = df.drop(['id', 'Station Description','Station Location', 'Latitude', 'Longitude'],axis=1) # axis=0 删除行，=1 删除列

#将Station用哑变量表示
dumm = pd.get_dummies(df[['Station Number']]) # '哑变量'矩阵
df = df.join(dumm)
del df['Station Number'] # 删除

#添加month,day,hour三个键和值
temp=pd.DatetimeIndex(df['Date and Time'])
df['date']=temp.date
df['time']=temp.time
df['hour']=pd.to_datetime(df.time,format="%H:%M:%S")#变换格式
df['hour']=pd.Index(df["hour"]).hour
df['month']=pd.to_datetime(df.date,format="%Y-%m-%d")#变换格式
df['month']=pd.Index(df["month"]).month
df['day']=pd.to_datetime(df.date,format="%Y-%m-%d")#变换格式
df['day']=pd.Index(df["day"]).day
df = df.drop(['Date and Time', 'date', 'time'],axis=1) # axis=0 删除行，=1 删除列
print(df.head(5))
print(df.shape)


X = np.array(df.drop(['Water_Level'], 1))
Y = np.array(df['Water_Level'])
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2)

training_features, testing_features, training_target, testing_target = \
            train_test_split(X, Y, random_state=None)

# Average CV score on the training set was:-0.043718068024817355
exported_pipeline = make_pipeline(
    StackingEstimator(estimator=ElasticNetCV(l1_ratio=0.7000000000000001, tol=1e-05)),
    StackingEstimator(estimator=DecisionTreeRegressor(max_depth=1, min_samples_leaf=2, min_samples_split=15)),
    SelectPercentile(score_func=f_regression, percentile=94),
    ExtraTreesRegressor(bootstrap=False, max_features=1.0, min_samples_leaf=2, min_samples_split=2, n_estimators=100)
)

exported_pipeline.fit(training_features, training_target)
results = exported_pipeline.predict(testing_features)
print("results: %s"%results)
score = exported_pipeline.score(testing_features, testing_target)
print("score: %s"%score)

同样，在后台运行:

nohup python -u pipeline.py > nohup.out 2>&1 &

输出如下:

results: [7.1012806  1.25462311 0.73743519 ... 2.12231535 4.22561277 1.27338528]
score: 0.9999991261258582

5. GPU

可惜的是，目前TPOT不支持GPU：
https://github.com/EpistasisLab/tpot/issues/680

不过，实际应用中我们发现，只是GPU-Util为0， GPU Memory Usage仍然可以达到10440MiB

屏幕快照 2019-01-30 下午6.32.55.png

6. 速度

TPOT在处理小规模数据的时候非常快，结果很给力。但处理大规模的数据问题，速度非常慢。所以可以尝试在数据清洗之后，抽样小部分数据跑一下TPOT，能得到一个还不错的算法。