timescaledb 集成 madlib

github 上有人提出了一个问题（2017 很早了），然后搜索timescaledb 的docs 文档，发现有
一片介绍的文章，所以尝试运行下
备注：环境使用虚拟机安装（没有使用docker madlib 的原因，实际上可以尝试基于timescaledb 的镜像改造）

安装madlib

这个可以参考madlib 官方文档，或者https://www.cnblogs.com/rongfengliang/p/10298159.html
因为timescaledb 是一个pg 的标准扩展，可以复用以前的安装方式

添加pg 10 repo

           
         yum install https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/pgdg-centos10-10-2.noarch.rpm

安装pg python 基本包

 
       yum -y install postgresql10-plpython supervisor

安装madlib 依赖包
注意python 版本,我使用python 2.7 安装时候失败了，修改为了python34

           
         yum update -y && yum install -y \ 
        
                             git \ 
        
                             gcc \ 
        
                             wget \ 
        
                             postgresql10-devel \ 
        
                             openssl \ 
        
                             m4 \ 
        
                             vim \ 
        
                             flex \ 
        
                             bison \ 
        
                             graphviz \ 
        
                             java \ 
        
                             epel-release \ 
        
                             python34-devel

安装pip 包
默认一般是包含的

           
         yum install -y python34-pip

pg_conf 配置(环境变量)

 
         PATH="$PATH:/usr/pgsql-10/bin"

安装python 依赖（通过pip）

 
          pip3 install awscli pygresql paramiko --upgrade

安装apache-madlib

 
         下载rpm 包 
        
         wget   https://dist.apache.org/repos/dist/release/madlib/1.15.1/apache-madlib-1.15.1-bin-Linux.rpm 
        
         安装 
        
         yum install -y apache-madlib-1.15.1-bin-Linux.rpm

timescaledb 扩展安装

下载timescaledb pg 扩展

 
         wget https://timescalereleases.blob.core.windows.net/rpm/timescaledb-1.1.1-postgresql-10-0.x86_64.rpm

安装

 
         yum install -y timescaledb-1.1.1-postgresql-10-0.x86_64.rpm

初始化数据库

 
         /usr/pgsql-10/bin/postgresql-10-setup initdb

配置pg 扩展
/var/lib/pgsql/10/data/postgresql.conf 文件

           
         + shared_preload_libraries = 'timescaledb'

修改pg_hba.conf 添加访问支持
之后修改之后，需要重启服务，systemctl restart postgresql-10

           
         /var/lib/pgsql/10/data/pg_hba.conf 
        
         修改如下： 
        
         # "local" is for Unix domain socket connections only 
        
         local all all trust 
        
         # IPv4 local connections: 
        
         host all all 127.0.0.1/32 trust 
        
         # IPv6 local connections: 
        
         host all all ::1/128 trust

启动数据库

 
         systemctl enable postgresql-10 
        
         systemctl start postgresql-10

注册madlib 扩展

下载&&安装madlib 扩展

 
         下载rpm 包 
        
         wget https://dist.apache.org/repos/dist/release/madlib/1.15.1/apache-madlib-1.15.1-bin-Linux.rpm 
        
         安装 
        
         yum install -y apache-madlib-1.15.1-bin-Linux.rpm

 
         /usr/local/madlib/bin/madpack -s madlib -p postgres -c postgres@localhost:5432/postgres install

效果

 
         madpack.py: INFO : Detected PostgreSQL version 10.6. 
        
         madpack.py: INFO : *** Installing MADlib *** 
        
         madpack.py: INFO : MADlib tools version = 1.15.1 (/usr/local/madlib/Versions/1.15.1/bin/../madpack/madpack.py) 
        
         madpack.py: INFO : MADlib database version = None (host=localhost:5432, db=postgres, schema=madlib) 
        
         madpack.py: INFO : Testing PL/Python environment... 
        
         madpack.py: INFO : > Creating language PL/Python... 
        
         madpack.py: INFO : > PL/Python environment OK (version: 2.7.5) 
        
         madpack.py: INFO : > Preparing objects for the following modules: 
        
         madpack.py: INFO : > - array_ops 
        
         madpack.py: INFO : > - bayes 
        
         madpack.py: INFO : > - crf 
        
         madpack.py: INFO : > - elastic_net 
        
         madpack.py: INFO : > - linalg 
        
         madpack.py: INFO : > - pmml 
        
         madpack.py: INFO : > - prob 
        
         madpack.py: INFO : > - sketch 
        
         madpack.py: INFO : > - svec 
        
         madpack.py: INFO : > - svm 
        
         madpack.py: INFO : > - tsa 
        
         madpack.py: INFO : > - stemmer 
        
         madpack.py: INFO : > - conjugate_gradient 
        
         madpack.py: INFO : > - knn 
        
         madpack.py: INFO : > - lda 
        
         madpac

检查

 
         /usr/local/madlib/bin/madpack -s madlib -p postgres -c postgres@localhost:5432/postgres install-check

效果

 
         TEST CASE RESULT|Module: bayes|bayes.ic.sql_in|PASS|Time: 121 milliseconds 
        
         TEST CASE RESULT|Module: crf|crf_train_small.ic.sql_in|PASS|Time: 112 milliseconds 
        
         TEST CASE RESULT|Module: crf|crf_test_small.ic.sql_in|PASS|Time: 133 milliseconds 
        
         TEST CASE RESULT|Module: elastic_net|elastic_net.ic.sql_in|PASS|Time: 133 milliseconds 
        
         TEST CASE RESULT|Module: linalg|linalg.ic.sql_in|PASS|Time: 44 milliseconds 
        
         TEST CASE RESULT|Module: linalg|svd.ic.sql_in|PASS|Time: 168 milliseconds 
        
         TEST CASE RESULT|Module: linalg|matrix_ops.ic.sql_in|PASS|Time: 238 milliseconds 
        
         TEST CASE RESULT|Module: prob|prob.ic.sql_in|PASS|Time: 21 milliseconds 
        
         TEST CASE RES

简单测试

内容来自官方文档
https://docs.timescale.com/v0.11/tutorials/tutorial-forecasting,https://docs.timescale.com/v1.1/tutorials/tutorial-hello-nyc
同时需要安装的组件也比较多，gis，timescale。。

预备安装
gis 扩展安装,后边需要,实际上这个是可选的

 
       wget https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/pgdg-centos10-10-2.noarch.rpm // 这个yum 源 
      
       上边实际上已经执行了 
      
       yum  install -y postgis25_10

加载扩展
都在postgres 数据库

           
         psql -U postgres -d postgres -h localhost  
        
         \c postgres 
        
         CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE; 
        
         CREATE EXTENSION postgis;

加载数据部分数据（因为gis 数据有依赖关系）
数据有点多，可能会比较慢

           
         wget https://timescaledata.blob.core.windows.net/datasets/nyc_data.tar.gz 
        
         tar -xvzf nyc_data.tar.gz 
        
         psql -U postgres -d postgres -h localhost < nyc_data.sql 
        
         psql -U postgres -d postgres -h localhost -c "\COPY rides FROM nyc_data_rides.csv CSV"

初始化gis 扩展 && forecast.sql
数据有点多，可能会比较慢

           
         ALTER TABLE rides ADD COLUMN pickup_geom geometry(POINT,2163); 
        
         ALTER TABLE rides ADD COLUMN dropoff_geom geometry(POINT,2163); 
        
         生成geo 数据： 
        
         UPDATE rides SET pickup_geom = ST_Transform(ST_SetSRID(ST_MakePoint(pickup_longitude,pickup_latitude),4326),2163); 
        
         UPDATE rides SET dropoff_geom = ST_Transform(ST_SetSRID(ST_MakePoint(dropoff_longitude,dropoff_latitude),4326),2163); 
        
         wget http://assets.iobeam.com/sql/forecast.sql 
        
         ​ 
        
         // 此过程会创建rides_count rides_length rides_price  等hypertables  
        
         psql -U postgres -d postgres -h localhost -f forecast.sql

查询操作

查询rides_price
如下：

           
         SELECT * FROM rides_price; 
        
               one_hour | trip_price 
        
         ---------------------+------------------ 
        
          2016-01-01 00:00:00 | 58.34 
        
          2016-01-01 01:00:00 | 58.34 
        
          2016-01-01 02:00:00 | 58.34 
        
          2016-01-01 03:00:00 | 58.34 
        
          2016-01-01 04:00:00 | 58.34 
        
          2016-01-01 05:00:00 | 59.59 
        
          2016-01-01 06:00:00 | 58.34 
        
          2016-01-01 07:00:00 | 60.3833333333333 
        
          2016-01-01 08:00:00 | 61.2575 
        
          2016-01-01 09:00:00 | 58.435 
        
          2016-01-01 10:00:00 | 63.952 
        
          2016-01-01 11:00:00 | 59.9576923076923 
        
          2016-01-01 12:00:00 | 60.4

创建训练以及测试的数据集
这个主要是测试从肯尼迪国际机场到时代广场的车程价格

           
         -- Make the training dataset 
        
         SELECT * INTO rides_price_train FROM rides_price 
        
         WHERE one_hour <= '2016-01-21 23:59:59'; 
        
         ​ 
        
         -- Make the testing dataset 
        
         SELECT * INTO rides_price_test FROM rides_price 
        
         WHERE one_hour >= '2016-01-22 00:00:00';

效果

 
         SELECT * INTO rides_price_train FROM rides_price 
        
         postgres-# WHERE one_hour <= '2016-01-21 23:59:59'; 
        
         SELECT 504 
        
         postgres=# SELECT * INTO rides_price_test FROM rides_price 
        
         postgres-# WHERE one_hour >= '2016-01-22 00:00:00'; 
        
         SELECT 240

使用madlib 函数生成训练数据

 
         DROP TABLE IF EXISTS rides_price_output; 
        
         DROP TABLE IF EXISTS rides_price_output_residual; 
        
         DROP TABLE IF EXISTS rides_price_output_summary; 
        
         DROP TABLE IF EXISTS rides_price_forecast_output; 
        
         ​ 
        
         SELECT madlib.arima_train('rides_price_train', -- input table 
        
               'rides_price_output', -- output table 
        
               'one_hour', -- timestamp column 
        
               'trip_price', -- time-series column 
        
               NULL, -- grouping columns 
        
               TRUE, -- include_mean 
        
               ARRAY[2,1,3] -- non-seasonal orders 
        
               ); 
        
         ​ 
        
         SELECT madlib.arima_forecast('rides_price_output', -- model table 
        
                                 'rides_price_forecast_output', -- output table 
        
                                 240 -- steps_ahead (10 days) 
        
                                 );

查看生成的结果

            
             SELECT * FROM rides_price_forecast_output;

数据

 
          SELECT * FROM rides_price_forecast_output; 
        
          steps_ahead | forecast_value 
        
         -------------+---------------- 
        
                    1 | 62.317574661 
        
                    2 | 62.7126520761 
        
                    3 | 62.892038632 
        
                    4 | 62.755044625 
        
                    5 | 62.6064068106 
        
                    6 | 62.6197088752 
        
                    7 | 62.7032172965 
        
                    8 | 62.729257786 
        
                    9 | 62.6956015739 
        
                   10 | 62.6685762986 
        
                   11 | 62.6755999496 
        
                   12 | 62.6926055721 
        
                   13 | 62.695637064 
        
                   14 | 62.6878845777 
        
                   15 | 62.683

结论：从肯尼迪国际机场到时代广场的车程价格在日常基础上保持不变

模型评估的

 
         ALTER TABLE rides_price_test ADD COLUMN id SERIAL PRIMARY KEY; 
        
         ALTER TABLE rides_price_test ADD COLUMN forecast DOUBLE PRECISION; 
        
         ​ 
        
         UPDATE rides_price_test 
        
         SET forecast = rides_price_forecast_output.forecast_value 
        
         FROM rides_price_forecast_output 
        
         WHERE rides_price_test.id = rides_price_forecast_output.steps_ahead; 
        
         ​ 
        
         SELECT madlib.mean_abs_perc_error('rides_price_test', 'rides_price_mean_abs_perc_error', 'trip_price', 'forecast'); 
        
         ​ 
        
         SELECT * FROM rides_price_mean_abs_perc_error;

结果

 
         SELECT * FROM rides_price_mean_abs_perc_error; 
        
          mean_abs_perc_error 
        
         --------------------- 
        
           0.0423789161947618

结论：从机场到曼哈顿的旅行价格保持在62美元，并且与测试数据集相比表现良好

说明

有一些关于机器学习以及统计的专业术语不是很懂，但是从基本的运行上，说明，madlib 与timescaledb 的集成还是
很方便的，这样我们既能有时序数据库的方便，同时还包含了基于sql 的机器学习能力，当然我们同时也可以集成graphql
engine 进行方便的graphql api ，总的来说基于标准pg 扩展的模型，部署还挺方便的

参考资料

https://docs.timescale.com/v0.11/tutorials/tutorial-forecasting
https://docs.timescale.com/v0.11/getting-started
https://docs.timescale.com/v1.1/tutorials/tutorial-hello-nyc#tutorial-postgis