mysql数据通过fluent同步到阿里云datahub填坑过程

gem sources -a https://ruby.taobao.org/这个步骤可能会报错:ERROR: While executing gem … (Gem::Exception)

Unable to require openssl, install OpenSSL and rebuild ruby (preferred) or use non-HTTPS sources

[root@Source ~]# gem sources -u ; gem source -l

ERROR: While executing gem … (Gem::Exception)

Unable to require openssl, install OpenSSL and rebuild ruby (preferred) or use non-HTTPS sources

解决这个问题需要

把/root/bi/ruby/lib/ruby/gems/2.4.0/gems/fluentd-0.14.13/fluent.conf拷贝到/et c/fluent目录

配置source和match

source配置:

match配置

这步可能会报/usr/local/lib/ruby/gems/2.2.0/gems/activerecord- 4.2.6/lib/active_record/connection_adapters/connection_specification.rb:177:in gem ‘mysql’` to your Gemfile (and ensure its version is at the minimum required by ActiveRecord). (Gem::LoadError)

这个问题是mysql插件需要用到mysql adapter适配器,需要安装mysql adapter适配器,执行以下命令:

注意是运行gem install mysql2 而不是gem install mysql,同时在source配置里面也要把adapter 设为mysql2。在这一步也可能遇到mysql包冲突的问题,所以建议在一个没有安装mysql的服务器上安装。

因为阿里云datahub是大小写敏感的,所以要把所有字段设为小写,不然字段会变为Null或空白。那么这需要怎么解决呢,查找fluent的官方文档,有record_transformer Filter Plugin这个东西,对于有大写字母的字段我们需配置一个filter,同时注意filter必须配置在match前面。

filter配置

猜你喜欢

转载自www.cnblogs.com/myjason2/p/11021310.html
今日推荐