分布式文件系统FastDFS+Nginx集成+FastDHF去重+SpringBoot文件长传与下载

概述

分布式⽂件系统（Distributed File System）是指⽂件系统管理的物理存储资源不⼀定直接连接在本地节点上，⽽是通过计算机⽹络与节点相连。

判断⼀个分布式⽂件系统是否优秀，取决于以下三个因素：
**1、数据的存储⽅式，**例如有1000万个数据⽂件，可以在⼀个节点存储全部数据⽂件，在其他N个节点上每个节点存储1000/N万个数据⽂件作为备份；或者平均分配到N个节点上存储，每个节点上存储1000/N万个数据⽂件。⽆论采取何种存储⽅式，⽬的都是为了保证数据的存储安全和⽅便获取。
**2、数据的读取速率，**包括响应⽤户读取数据⽂件的请求、定位数据⽂件所在的节点、读取实际硬盘中数据⽂件的时间、不同节点间的数据传输时间以及⼀部分处理器的处理时间等。各种因素决定了分布式⽂件系统的⽤户体验。即分布式⽂件系统中数据的读取速率不能与本地⽂件系统中数据的读取速率相差太⼤，否则在本地⽂件系统中打开⼀个⽂件需要2秒，⽽在分布式⽂件系统中各种因素的影响下⽤时超过10秒，就会严重影响⽤户的使⽤体验。
**3、数据的安全机制，**由于数据分散在各个节点中，必须要采取冗余、备份、镜像等⽅式保证节点出现故障的情况下，能够进⾏数据的恢复，确保数据安全。

文件系统分类：

• 块存储：MongoDB数据库中的GridFS、Hadoop中的HDFS，这些系统在存储⽂件的的时候会尝试先将⽂件打碎存储（拆分成Data Block）。这样存储的优点可以存储超⼤型⽂件，更加⾼效的利⽤磁盘资源。但是需要额外存储⽂件碎⽚的元数据信息。

• ⽂件存储:GlusterFS、NFS、FastDFS等都是基于⽂件单位存储，这种存储并不会将⽂件系统打碎。⽽是⽂件存储到系统中的某⼀台服务器中。这样存储的优点可以应对⼀些⼩⽂件系统，系统维护简单，⽆需存储⽂件的元数据，系统设计和维护成本低。

FastDFS介绍

特点：
FastDFS 是⼀款开源的轻量级分布式⽂件系统如下特点：

• 纯粹C语⾔实现，⽀持Linux、FreeBSD等unix系统。
• 类似GoogleFS/HDFS，但是不是通⽤的⽂件系统，只能通过专有的API访问，⽬前提供了C、Java和PHPAPI 互联⽹量身定做，最求⾼性能，⾼扩展.
• FastDFS不仅仅可以存储⽂件，还可以存储⽂件的元数据信息（可选）。
  架构
  整个FastDFS架构中有Client、Tracker和Storage服务。其中Client⽤于提交⽂件给FastDFS集群。Storage 服务负责实际数据的存储，Tracker服务负责监控和调度Storage服务，起到负载均衡器的作⽤。
  
  如果Storage的中 卷 是⼀样的也就意味着这些服务彼此数据相互备份，实现数据的冗余备份。不同 卷存储整个集群中的部分⽂件，类似于传统单机⽂件系统的的分区概念（C盘、D盘、…）Tracker Server 主要做⼯作调度，在访问上起负载均衡的作⽤。在内存中记录集群中group/ 卷 和Storage server的状态信息，是连接Client和Storage Server的枢纽。因为相关信息存储在内存中，所以Tracker Server性能⾮常⾼,⼀个较⼤的集群（上百个group/ 卷 ）中3台就够了。

StorageServer ：存储服务器，⽂件和⽂件属性信息（meta数据）都存储在服务器的磁盘上。
上传、下载机制



FastDFS集群搭建
安装准备工作：
1.三台虚拟机

 1. 10.15.0.29 master
 2. 10.15.0.30 slave1
 3. 10.15.0.31 slave2

2.资源下载安装
进⼊ https://github.com/happyfish100 ⽹站下载FastDFS相关资源。建议使⽤⼩编给⼤家处理过的安装包。

1. 安装依赖包libfastcommonhttps://github.com/happyfish100/libfastcommon/archive/V1.0.35.tar.gz

[root@CentOSX ~]# yum install -y gcc-c++
[root@CentOSX ~]# tar -zxf V1.0.35.tar.gz
[root@CentOSX ~]# cd libfastcommon-1.0.35
[root@CentOSX libfastcommon-1.0.35]# ./make.sh
[root@CentOSX libfastcommon-1.0.35]# ./make.sh install

2. 安装FastDFS https://github.com/happyfish100/fastdfs/archive/V5.11.tar.gz

[root@CentOSX ~]# yum install -y perl-devel
[root@CentOSX ~]# tar -zxf fastdfs-5.11.tar.gz
[root@CentOSX ~]# cd fastdfs-5.11
[root@CentOSX fastdfs-5.11]# ./make.sh
[root@CentOSX fastdfs-5.11]# ./make.sh install

提示:当软件安装结束后，默认FastDFS启动所需的配置⽂件放置在/etc/fdfs⽬录下。

[root@CentOSX ~]# yum install -y tree
[root@CentOSX ~]# tree /etc/fdfs/
/etc/fdfs/
[root@CentOX ~]#  tree /etc/fdfs/
/etc/fdfs/
├── client.conf.sample
├── storage.conf.sample
├── storage_ids.conf.sample
└── tracker.conf.sample

0 directories, 4 files

# 可运⾏脚本
[root@CentOSX ~]# ls -l /etc/init.d/fdfs_*
-rwxr-xr-x. 1 root root 961 Jun 30 20:22 /etc/init.d/fdfs_storaged
-rwxr-xr-x. 1 root root 963 Jun 30 20:22 /etc/init.d/fdfs_trackerd
# 执⾏程序
[root@CentOSX ~]# whereis fdfs_storaged fdfs_trackerd
fdfs_storaged: /usr/bin/fdfs_storaged
fdfs_trackerd: /usr/bin/fdfs_trackerd

3. 配置服务

• 1.创建fdfs运⾏所需的数据⽬录

[root@CentOSX ~]# mkdir -p /data/fdfs/{tracker,storage/store01,storage/store02}
[root@CentOSX ~]# tree /data/
/data/
/data/
└── fdfs
    ├── storage
    │   ├── store01
    │   └── store02
    └── tracker

• 2.创建启动所需的配置⽂件

[root@CentOSX ~]# cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf
[root@CentOSX ~]# cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf
[root@CentOSX ~]# cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf
[root@CentOSX ~]# tree /etc/fdfs/
/etc/fdfs/
├── client.conf
├── client.conf.sample
├── storage.conf
├── storage.conf.sample
├── storage_ids.conf.sample
├── tracker.conf
└── tracker.conf.sample

• 3.配置Tracker Server

[root@CentOSX ~]# vi /etc/fdfs/tracker.conf

base_path=/data/fdfs/tracker

• 4.配置Storage Server

[root@CentOSX ~]# vi /etc/fdfs/storage.conf

group_name=group`[1,2,3]`
base_path=/data/fdfs/storage
store_path_count=2
store_path0=/data/fdfs/storage/store01
store_path1=/data/fdfs/storage/store02
tracker_server=CentOSA:22122
tracker_server=CentOSB:22122
tracker_server=CentOSC:22122

• 5.修改Client端

[root@CentOSX ~]# vi /etc/fdfs/client.conf

base_path=/tmp
tracker_server=CentOSA:22122
tracker_server=CentOSB:22122
tracker_server=CentOSC:22122

• 启动服务器

[root@CentOSX ~]# /etc/init.d/fdfs_trackerd start|stop|restart
Starting FastDFS tracker server:
[root@CentOSX ~]# /etc/init.d/fdfs_storaged start|stop|restart
Starting FastDFS storage server:
[root@CentOSX ~]# ps -aux | grep fdfs
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root 78950 0.0 0.1 144784 2040 ? Sl 21:06 0:00
/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
root 79000 13.0 3.2 83520 67144 ? Sl 21:06 0:06
/usr/bin/fdfs_storaged /etc/fdfs/storage.conf
root 79324 0.0 0.0 103320 884 pts/0 S+ 21:07 0:00 grep fdfs

FastDFS操作

• 上传文件

在这里插入代码片

 1. [root@CentOSX ~]# fdfs_upload_file /etc/fdfs/client.conf install.log
group1/M00/00/01/wKikgV0YtQ2AfmozAAAixZ60QfI792.log

• 下载文件

[root@CentOSX ~]# fdfs_download_file /etc/fdfs/client.conf
group1/M00/00/01/wKikgV0YtQ2AfmozAAAixZ60QfI792.log

• 查询信息

[root@CentOSX ~]# fdfs_file_info /etc/fdfs/client.conf
group1/M00/00/01/wKikgV0YtQ2AfmozAAAixZ60QfI792.log
source storage id: 0
source ip address: 192.168.164.129
file create timestamp: 2019-06-30 21:11:41
file size: 8901
file crc32: 2662613490 (0x9EB441F2)

• 删除文件

[root@CentOSX ~]# fdfs_delete_file /etc/fdfs/client.conf
group1/M00/00/01/wKikgV0YtQ2AfmozAAAixZ60QfI792.log

• ⽂件追加

[root@CentOSX ~]# clear
[root@CentOSX ~]# echo "hello" > 1.txt
[root@CentOSX ~]# echo "word" > 2.txt
[root@CentOSX ~]# fdfs_upload_appender /etc/fdfs/client.conf /root/1.txt
group2/M00/00/00/wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt
[root@CentOSX ~]# fdfs_append_file /etc/fdfs/client.conf
group2/M00/00/00/wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt 2.txt
[root@CentOSX ~]# fdfs_download_file /etc/fdfs/client.conf
group2/M00/00/00/wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt
[root@CentOSX ~]# cat wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt
hello
word

• 监视状态

[root@CentOSX ~]# fdfs_monitor /etc/fdfs/client.conf

• ⽂件校验和

[root@CentOSX ~]# fdfs_crc32 /etc/fdfs/client.conf
group2/M00/00/00/wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt
2911662598

Nginx集成FastDFS

Nginx配置安装

• 下载fastdfs-nginx-module（不建议使⽤github上，因为编译有问题）

[root@CentOSX ~]# tar -zxf fastdfs-nginx-module.tar.gz
[root@CentOSX ~]# yum install -y pcre-devel
[root@CentOSX ~]# yum install -y openssl-devel
[root@CentOSX ~]# wget http://nginx.org/download/nginx-1.11.1.tar.gz
[root@CentOSX ~]# tar -zxf nginx-1.11.1.tar.gz
[root@CentOSX nginx-1.11.1]# ./configure --prefix=/usr/local/nginx-1.11.1/ --addmodule=/root/fastdfs-nginx-module/src
[root@CentOSX nginx-1.11.1]# make
[root@CentOSX nginx-1.11.1]# make install

• 拷⻉配置

[root@CentOSX ~]# cp /root/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/
[root@CentOSX ~]# cd /root/fastdfs-5.11/conf/
[root@CentOSX conf]# cp http.conf mime.types anti-steal.jpg /etc/fdfs/

• 配置nginx.conf

[root@CentOSX ~]# vi /usr/local/nginx-1.11.1/conf/nginx.conf

#access_log logs/host.access.log main;

 location ~ /group[0-9]+/M00 {
 root /data/fdfs/storage/store01;
 ngx_fastdfs_module;
 }
 location ~ /group[0-9]+/M01 {
 root /data/fdfs/storage/store02;
 ngx_fastdfs_module;
 }
 
 location / {
 root html;
 index index.html index.htm;
 }      

• 修改mod_fastdfs.conf

[root@CentOSX ~]# vi /etc/fdfs/mod_fastdfs.conf

tracker_server=CentOSA:22122
tracker_server=CentOSB:22122
tracker_server=CentOSC:22122
group_name=group`[1,2,3]`
url_have_group_name = true
store_path_count=2
store_path0=/data/fdfs/storage/store01
store_path1=/data/fdfs/storage/store02

• 启动nginx

[root@CentOSX ~]# cd /usr/local/nginx-1.11.1/
[root@CentOSX nginx-1.11.1]# ./sbin/nginx -t
ngx_http_fastdfs_set pid=116305
ngx_http_fastdfs_set pid=116305
nginx: the configuration file /usr/local/nginx-1.11.1//conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx-1.11.1//conf/nginx.conf test is successful
[root@CentOSX nginx-1.11.1]# ./sbin/nginx

• 测试下载

[root@CentOSA ~]# fdfs_upload_file /etc/fdfs/client.conf install.log
group2/M01/00/00/wKikgl0p_4-AIqGJAAAixZ60QfI693.log

访问⼀个nginx服务查看效果

http://CentOS[A|B|C]/group2/M01/00/00/wKikgl0p_4-AIqGJAAAixZ60QfI693.log?
filename=install.log

⽤户在请求的时候，可以选择性添加⽂件名，⽤于修改下载的⽂件名

FastDHT⽂件去重

FastDFS除了提供了与nginx的集成，已提供了去重的⽂件解决⽅案。该解决⽅案FastDFS的作者yuqing也 在github上以FastDHT分⽀贡献出来了。
FastDHT is a high performance distributed hash table (DHT) which based key value pairs. It can store mass
key value pairs such as filename mapping, session data and user related data.

安装

• 1.安装BerkeleyDB 下载db-4.7.25.tar.gz

[root@CentOSX ~]# tar -zxf db-4.7.25.tar.gz
[root@CentOSX ~]# cd db-4.7.25
[root@CentOSX db-4.7.25]# cd build_unix/
[root@CentOSX build_unix]# ./../dist/configure
[root@CentOSX build_unix]# make
[root@CentOSX build_unix]# make install

• 2.安装FastDHT

[root@CentOSX ~]# tar zxf FastDHT_v2.01.tar.gz
[root@CentOSX ~]# cd FastDHT
[root@CentOSX FastDHT]# ./make.sh
[root@CentOSX FastDHT]# ./make.sh install

安装结束后会在/etc⽬录下产⽣fdht⽂件夹

[root@CentOSX fdht]# tree /etc/fdht/
/etc/fdht/
├── fdht_client.conf
├── fdhtd.conf
└── fdht_servers.conf

• 3.修改fdhtd.conf

[root@CentOSX ~]# mkdir /data/fastdht
[root@CentOSX ~]# vi /etc/fdht/fdhtd.conf
base_path=/data/fastdht

• 4.修改fdht_servers.conf

[root@CentOSX ~]# vi /etc/fdht/fdht_servers.conf
group_count = 3
group0 = CentOSA:11411
group1 = CentOSB:11411
group2 = CentOSC:11411

• 5.修改fdht_client.conf配置⽂件

[root@CentOSX ~]# vi /etc/fdht/fdht_client.conf
base_path=/tmp/

• 6.启动FDHT服务

[root@CentOSX ~]# fdhtd /etc/fdht/fdhtd.conf start|stop|restart
[root@CentOSX ~]# ps -axu| grep fd
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root 29127 0.0 0.8 195200 17504 ? Sl 00:38 0:00 fdhtd
/etc/fdht/fdhtd.conf start
root 29193 0.0 0.0 103320 884 pts/0 S+ 00:39 0:00 grep fd
root 128672 0.2 0.1 275856 2204 ? Sl 00:14 0:03
/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
root 128710 1.8 3.2 85584 67188 ? Sl 00:14 0:27
/usr/bin/fdfs_storaged /etc/fdfs/storage.conf

操作FastDHT服务

• 设置值

[root@CentOSX ~]# fdht_set /etc/fdht/fdht_client.conf jiangzz:user001
name='jiangzz',age=18;
This is FastDHT client test program v2.01
Copyright (C) 2008, Happy Fish / YuQing
FastDHT may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDHT source kit.
Please visit the FastDHT Home Page http://www.csource.org/
for more detail.
success set key count: 2, fail count: 0

• 读取值

[root@CentOSX ~]# fdht_get /etc/fdht/fdht_client.conf jiangzz:user001 name,age
This is FastDHT client test program v2.01
Copyright (C) 2008, Happy Fish / YuQing
FastDHT may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDHT source kit.
Please visit the FastDHT Home Page http://www.csource.org/
for more detail.
name=jiangzz
age=18
success get key count: 2, fail count: 0

• 删除值

[root@CentOSX ~]# fdht_delete /etc/fdht/fdht_client.conf jiangzz:user001 name;
This is FastDHT client test program v2.01
Copyright (C) 2008, Happy Fish / YuQing
FastDHT may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDHT source kit.
Please visit the FastDHT Home Page http://www.csource.org/
for more detail.
success delete keys: name
success delete key count: 1, fail count: 0

集成FastDHT

• 1.修改etc/fdfs/storage.conf配置⽂件

[root@CentOSX ~]# vi /etc/fdfs/storage.conf
check_file_duplicate=1
keep_alive=1
#include /etc/fdht/fdhtservers.conf

• 2.分别启动fdhtd服务、fastfs

[root@CentOSX usr]# /usr/local/bin/fdhtd /etc/fdht/fdhtd.conf restart
[root@CentOSX usr]# /etc/init.d/fdfs_trackerd restart
[root@CentOSX usr]# /etc/init.d/fdfs_storaged restart

• 3.上产⽂件测试

[root@CentOSA ~]# fdfs_upload_file /etc/fdfs/client.conf install.log
group2/M00/00/00/wKikgl0qC4KAErBTAAAixXWAIyY133.log
[root@CentOSA ~]# fdfs_upload_file /etc/fdfs/client.conf install.log
group2/M00/00/00/wKikgl0qDAqAa0XwAAAixWB5m1c851.log


[root@CentOSB ~]# ls -l /data/fdfs/storage/store01/data/00/00/total 20
lrwxrwxrwx. 1 root root 72 Jul 14 00:49 wKikgl0qC4KAErBTAAAixXWAIyY133.log ->
/data/fdfs/storage/store01/data/00/00/wKikgl0qC4KARrYBAAAixZ60QfI755.log
-rw-r--r--. 1 root root 8901 Jul 14 00:49 wKikgl0qC4KARrYBAAAixZ60QfI755.log
lrwxrwxrwx. 1 root root 72 Jul 14 00:51 wKikgl0qDAqAa0XwAAAixWB5m1c851.log ->
/data/fdfs/storage/store01/data/00/00/wKikgl0qC4KARrYBAAAixZ60QfI755.log

可以看出系统产⽣了wKikgl0qC4KAErBTAAAixXWAIyY133.log的两个链接

SpringBoot集成FastDFS

引⼊依赖

<dependency>
 <groupId>com.github.tobato</groupId>
 <artifactId>fastdfs-client</artifactId>
 <version>1.26.6</version>
</dependency>

yml配置文件

#文件系统
fdfs:
  tracker-list: pro1:22122,pro2:22122,pro3:2122
  # 配置默认缩略图
  thumb-image:
    height: 80
    width: 100

测试⽂件上传

@Autowired
private FastFileStorageClient fastFileStorageClient;

• ⽂件上传

  
    public void wjUpload() throws IOException {
        FileInputStream inputStream = new FileInputStream("C:\\Users\\finejade\\aa/hadoop.jpg");
        FastImageFile fastImageFile=new
                FastImageFile(inputStream,inputStream.available(),"jpg",new HashSet<MetaData>());
        StorePath storePath = fastFileStorageClient.uploadImage(fastImageFile);
        System.out.println(storePath.getFullPath());
    }

• 图⽚上传（缩略图）

   
    public void imageUpload() throws IOException {
        FileInputStream inputStream = new FileInputStream("C:\\Users\\finejade\\aa/hadoop.jpg");
        FastImageFile fastImageFile=new
                FastImageFile(inputStream,inputStream.available(),"jpg",new HashSet<MetaData>(),new
                ThumbImage(150,150));
        StorePath storePath = fastFileStorageClient.uploadImage(fastImageFile);
        System.out.println(storePath.getFullPath());
    }

• 删除⽂件

fastFileStorageClient.deleteFile("group3","M00/00/00/ygthrl0prl6AX6ygAL26hh1kYdE312_80
x80.png");

• ⽂件下载(⽤不到)

ByteArrayOutputStream baos = fastFileStorageClient.downloadFile("group3",
"M00/00/00/wKjvgl0prTSAMjeTAL26hhzQmiQ959.png", new
DownloadCallback<ByteArrayOutputStream>() {
 @Override
 public ByteArrayOutputStream recv(InputStream ins) throws IOException {
 ByteArrayOutputStream baos = new ByteArrayOutputStream();
 IOUtils.copy(ins, baos);
 return baos;
 }
 });
IOUtils.copy(new ByteArrayInputStream(baos.toByteArray()),new FileOutputStream("G:/素材
资料/baby.png"));