File management is an indispensable part of every part, especially for some photo album websites and video websites, file management services are a very important part. There are many open source distributed file systems, including HDFS, MooseFS, FastDFS, etc. Each distributed system has its own characteristics, but the general functions include: file storage, file access, file synchronization, and some systems will help us solve Mass storage and load balancing issues. Among the various distributed systems, I only know about HDFS and FastDFS. The focus of the two is different. HDFS is suitable for storing and managing large files, but not suitable for storing a large number of small files, frequently modifying files and a large number of random files. Read, HDFS is often used in the field of big data analysis, and FastDFS, on the contrary, is especially suitable for storing and managing small files. So here is an introduction to FastDFS.
There are three important roles in FastDFS: Tracker Server, Storage Server, Client.
Tracker Server: Tracking server, responsible for file management, scheduling, control center and load balancing operations. It is especially similar to the registration center [equivalent to Dubbo], so each Storage Server will connect to the Tracker Server after startup, inform itself of its information, and keep a heartbeat.
Storage Server: The storage server, which is a role that really works in FastDFS, provides services for file upload/download, file modification, and file deletion. In the Storage Server, the unit is Group. There can be several Storage Servers in each Group, and data backup can be continuously realized between several Storage Servers.
Client: The client, that is, our own server, calls FastDFS to implement operations such as file upload and download.
File upload process
1) Storage Server will regularly upload information to Tracker Server and register in Tracker Server.
2) The upload connection request will first access the Tracker Server, and the Tracker Server will return an idle and available Storage Server. Return the IP and port number of the Storage Server.
3) After the Client gets the IP and port number returned by the Tracker Server, it starts uploading the file.
4) Storage Server writes the file into the local disk of the server, and returns the storage information of the file to the user. Uploaded files generate a unique ID value.
After the upload is completed, a file ID will be returned to the client, which is used to access the index information of the file in the future. The index information of the file includes: group name, virtual disk path, two-level data directory, and file name.
For example, return: group1 /M00 /02/44/wKgDrE34E8wAAAAAAAAAGkEIYJK42378.sh .
Group name [ group1 ]: The name of the storage group where the file is uploaded. After the file is uploaded successfully, the storage server will return it, and the client needs to save it by itself.
Virtual disk path [ /M00 ]: The virtual path of storage configuration, which corresponds to the disk option store_path*. If store_path0 is configured, it is M00, if store_path1 is configured, it is M01, and so on.
Two-level directory for data [ /02/44/ ]: A two-level directory created by the storage server under each virtual disk path for storing data files.
File name [ wKgDrE34E8wAAAAAAAAAGkEIYJK42378.sh ]: It is different from the file uploaded. It is generated by the storage server based on specific information, and the file name includes: source storage server IP address, file creation timestamp, file size, random number, file extension and other information.
File download process
1) Client sends a download request to Tracker Server, and Tracker Server returns the Storage Server where the corresponding storage file is located, and then returns its IP and port.
2) The Client directly establishes a connection with one of the Storage Servers through the returned IP and port and downloads the specified file.
The construction of file service
FastDFS can be installed using Docker. You can refer to this article " Docker Install FastDFS ". The file service here uses a microservice based on SpringClound.
① Prepare the configuration file of FastDFS
connect_timeout=60
network_timeout=60
charset=UTF-8
# Tracker的Http请求端口
http.tracker_http_port=8080
# Tracker的TCP通信端口
tracker_server=192.168.132.132:22122② Project yml file configuration
spring:
servlet:
multipart:
max-file-size: 10MB
max-request-size: 10MB
application:
name: file
server:
port: 18082
eureka:
client:
service-url:
defaultZone: http://127.0.0.1:7001/eureka
instance:
prefer-ip-address: true
feign:
hystrix:
enabled: true③Introduce the corresponding maven dependencies
<dependency>
<groupId>net.oschina.zcx7878</groupId>
<artifactId>fastdfs-client-java</artifactId>
<version>1.27.0.0</version>
</dependency>④Create a corresponding FastDFS tool class
/**
* 实现FastDfs文件管理：
* 文件上传
* 文件删除
* 文件下载
* 文件信息获取
* Storage信息获取
* Tracker信息获取
*
* @author SunRains
* @date 2021/3/2 0002
*/
public class FastDFSUtil {
static {
try {
String item = new ClassPathResource("item").getPath();
System.out.println(item);
String fileName = new ClassPathResource("fdfs_client.conf").getPath();
ClientGlobal.init(fileName);
} catch (IOException e) {
e.printStackTrace();
} catch (MyException e) {
e.printStackTrace();
}
}
/**
* 文件上传
* @param fastDFSFile :上传文件信息封装
*/
public static String[] upload(FastDFSFile fastDFSFile) throws Exception {
NameValuePair[] metaList=new NameValuePair[1];
metaList[0]=new NameValuePair("author",fastDFSFile.getAuthor());
// 通过StorageClient访问Storage，实现文件上传，并且获取文件上传后的存储信息
StorageClient storageClient = getStorageClient();
/*
通过storageClient访问Storage，实现文件上传，并且获取文件上传的存储信息
1：上传文件的字节数组
2：文件的扩展名：jpg
3：附加参数：比如 拍摄地址：北京
返回数组
[0]:文件上传所存储的Storage的组名字 group1
[1]:文件存储到Storage上的文件名字 /M00/02/04/xx.jpg包括虚拟路径
*/
return storageClient.upload_file(fastDFSFile.getContent(), fastDFSFile.getExt(), metaList);
}
/**
* 获取文件信息
* @param groupName 文件组名:group1
* @param remoteFileName 文件的存储路径:M00/00/00/wKiEhGA-y1-AB3p8AABhjgoR8Vw160.png
*/
public static FileInfo getFile(String groupName, String remoteFileName) throws Exception {
// 创建TrackerClient对象，通过TrackerClient访问TrackerServer
StorageClient storageClient = getStorageClient();
// 获取文件信息
return storageClient.get_file_info(groupName,remoteFileName);
}
/**
* 文件下载
* @param groupName 文件组名:group1
* @param remoteFileName 文件的存储路径:M00/00/00/wKiEhGA-y1-AB3p8AABhjgoR8Vw160.png
*/
public static InputStream downloadFile(String groupName, String remoteFileName) throws Exception{
// 创建TrackerClient对象，通过TrackerClient访问TrackerServer
StorageClient storageClient = getStorageClient();
byte[] buffer = storageClient.download_file(groupName, remoteFileName);
return new ByteArrayInputStream(buffer);
}
/**
* 删除文件
* @param groupName 文件组名:group1
* @param remoteFileName 文件的存储路径:M00/00/00/wKiEhGA-y1-AB3p8AABhjgoR8Vw160.png
*/
public void deleteFile(String groupName, String remoteFileName) throws Exception{
StorageClient storageClient = getStorageClient();
storageClient.delete_file(groupName,remoteFileName);
}
/**
* 获取StorageClient对象
* @return
* @throws IOException
*/
private static StorageClient getStorageClient() throws IOException {
// 创建TrackerClient对象，通过TrackerClient访问TrackerServer
TrackerClient trackerClient = new TrackerClient();
// 通过TrackerClient获取TrackerServer的连接对象
TrackerServer trackerServer = trackerClient.getConnection();
// 通过TrackerServer获取Storage信息，创建StorageClient对象存储Storage信息
return new StorageClient(trackerServer, null);
}
/**
* 获取Storage信息
* @throws Exception
*/
public static StorageServer getStorage() throws Exception{
// 创建TrackerClient对象，通过TrackerClient访问TrackerServer
TrackerClient trackerClient = new TrackerClient();
// 通过TrackerClient获取TrackerServer的连接对象
TrackerServer trackerServer = trackerClient.getConnection();
// 获取Storage信息
return trackerClient.getStoreStorage(trackerServer);
}
/**
* 获取Tracker信息
* @throws Exception
*/
public static String getTrackerInfo() throws Exception{
// 创建TrackerClient对象，通过TrackerClient访问TrackerServer
TrackerClient trackerClient = new TrackerClient();
// 通过TrackerClient获取TrackerServer的连接对象
TrackerServer trackerServer = trackerClient.getConnection();
// Tracker的IP和HTTP端口
String ip = trackerServer.getInetSocketAddress().getHostString();
int tracker_http_port = ClientGlobal.getG_tracker_http_port();
return "http://"+ip+":"+tracker_http_port;
}
/**
* 获取Storage的IP和端口信息
*/
public static ServerInfo[] getServerInfo(String groupName, String remoteFileName) throws Exception{
// 创建TrackerClient对象，通过TrackerClient访问TrackerServer
TrackerClient trackerClient = new TrackerClient();
// 通过TrackerClient获取TrackerServer的连接对象
TrackerServer trackerServer = trackerClient.getConnection();
// 获取Storage的IP和端口信息
return trackerClient.getFetchStorages(trackerServer,groupName,remoteFileName);
}
public static void main(String[] args) throws Exception {
// 文件下载
InputStream inputStream = downloadFile("group1", "M00/00/00/wKiEhGA-y1-AB3p8AABhjgoR8Vw160.png");
// 将文件写入本地磁盘
FileOutputStream os=new FileOutputStream("D:/1.png");
// 定义一个缓冲区
byte[] buffer=new byte[1024];
while(inputStream.read(buffer)!=-1){
os.write(buffer);
}
os.flush();
os.close();
inputStream.close();
}
}
Of course, you need to encapsulate a file information object FastDFSFile yourself.
public class FastDFSFile implements Serializable {
//文件名字
private String name;
//文件内容
private byte[] content;
//文件扩展名
private String ext;
//文件MD5摘要值
private String md5;
//文件创建作者
private String author;
public FastDFSFile(String name, byte[] content, String ext, String md5, String author) {
this.name = name;
this.content = content;
this.ext = ext;
this.md5 = md5;
this.author = author;
}
public FastDFSFile(String name, byte[] content, String ext) {
this.name = name;
this.content = content;
this.ext = ext;
}
public FastDFSFile() {
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public byte[] getContent() {
return content;
}
public void setContent(byte[] content) {
this.content = content;
}
public String getExt() {
return ext;
}
public void setExt(String ext) {
this.ext = ext;
}
public String getMd5() {
return md5;
}
public void setMd5(String md5) {
this.md5 = md5;
}
public String getAuthor() {
return author;
}
public void setAuthor(String author) {
this.author = author;
}
@Override
public String toString() {
return "FastDFSFile{" +
"name='" + name + '\'' +
", content=" + Arrays.toString(content) +
", ext='" + ext + '\'' +
", md5='" + md5 + '\'' +
", author='" + author + '\'' +
'}';
}
}
⑤ Write the corresponding Controller.
/**
* @author SunRains
* @date 2021/3/2 0002
*/
@RestController
@CrossOrigin
@RequestMapping(value = "/upload")
public class FileUploadController {
@PostMapping
public Result upload(@RequestParam(value = "file")MultipartFile file) throws Exception{
// 封装文件信息
FastDFSFile fastDFSFile=new FastDFSFile(file.getOriginalFilename(),
file.getBytes(), StringUtils.getFilenameExtension(file.getOriginalFilename()));
String[] upload = FastDFSUtil.upload(fastDFSFile);
// 拼接访问地址
String url="http://192.168.132.132:8080/"+upload[0]+"/"+upload[1];
return new Result<>(true, StatusCode.OK,"上传文件成功",url);
}
}
In this way, if other modules need to upload files in the future, they only need to call this interface to realize the file upload operation.