Article Directory
Understand the concept:
Fragmentation: Files are transferred multiple times, a small part is transferred each time, and all files are merged at the end
Second transmission: first send a simple request to let the backend determine whether it exists, if it exists, it will not send it directly, if it does not exist, then make a second request, and the second request contains the file stream. Avoid wasting bandwidth in transmission of large files during duplication.
Continued upload: On the basis of the orderly uploading of fragments, we will only perform one inspection. The backend finds the latest fragment (with the largest number) and returns the fragment number. The front end receives the fragment number and uploads it to the uploader. The index is set to the fragment number + 1 and then continue to retransmit.
This article first simply implements fragment upload and ordinary second transmission. The http concurrent number is set to 1 to ensure that the fragments are transmitted in order. As for the fragment verification, the complete file verification is left in the next article.
找到的比较优秀的demo博客：
https://www.cnblogs.com/xiahj/p/vue-simple-uploader.html#file%E7%9A%84%E5%8A%A8%E6%80%81params%E7%BB%91%E5%AE%9A
1. Front end
For the introduction of the parameters and events required by the component, go to the vue-simple-uploader document I compiled, which is a wheel that github is looking for.
[uploader] Tabular self-organization of vue-simple-uploader documents (super detailed)
Let's clarify the process first:
1. Click the uploader, after selecting the file, we use automatic upload, that is, upload after selecting
2. Judgment + send.
The uploader tests the selected file first, that is, it tries to send it. It does not contain the file stream, but only some common values, which will involve the testMethod, testChunks and checkChunkUploadedByResponse in the component parameters. You can pay attention to it a little bit . That is to say, a file will be sent twice. The first request does not contain the file, so that the backend can determine whether the file or fragment exists. If the backend returns true to indicate that the file already exists, the file will not be sent, otherwise the second request will be sent. The file stream is included in the secondary request.
I demonstrate with a request that will fail:
As you can see, we only have 3 shards, but 6 requests are sent, and we can see that the red request does not contain the file field
And this successfully sent request contains the file field, which shows that the uploader sends a simple request by default to determine whether the file already exists, so as to realize the instant transmission of uploaded duplicate files, and avoid files being occupied in the network when they are repeated . bandwidth.
3. Sequential upload of multipart upload
Our front end has set the concurrency number to 1, and there will be no loss of intermediate fragments, because the fragments that fail to upload will be retransmitted, and if the retransmission fails, it will stop and will not say skip.
However, if you set a high concurrency number, the subsequent fragments will arrive first.
For high concurrency, we will try again in the next article
4. You should not check every shard
In the third point, on the basis of order, our check should be enableOnceCheck, that is, only one check will be performed. The backend finds the latest shard (with the largest number) and returns the shard number. The front end receives the shard segment number and set the index of the uploader to the segment number + 1 and continue to retransmit.
1.1 uploader component
- Consider the uploader as a whole uploader, uploader-btn is the button we see, click the button to trigger the uploader.
<uploader
:options="this.options"
@file-added="this.fileAdded"
style="margin-right: 30px;float: left;">
<uploader-unsupport></uploader-unsupport>
<uploader-btn class="uploader-btn">
点击上传
</uploader-btn>
</uploader>
- Among them, we bind options, that is, the configuration parameters of the uploader. Please refer to the documentation for specific parameters.
data() {
return {
options: {
target: "/file/uploadFile", //目标位置，后端的位置
//query是额外的参数，属于query式参数，后端要用@RequestParam接收
query: {
curUrl: this.$store.state.file.curUrl },
//testMethod和uploadMethod的请求方法设置为不同，方便后端接口编写
testMethod: "GET", //这里表示的是秒传的优先判断的请求方式
uploadMethod: "POST", //这里表示的是发送文件流的请求方式
//单个分片的大小，这里表示5Mb
chunkSize: 5 * 1024 * 1024,
forceChunkSize: true, //强制每个分片都小于chunkSize，否则可能出现单个分片过大的情况
testChunks: true,//是否开启服务器分片校验，开启就是可以实现秒传
//一个回调函数，也就是第一次不带文件的请求的一个回调，一般后端对应接口需要返回skipUpload【true or false】
//js中一个函数是可以作为值的
checkChunkUploadedByResponse:
function (chunk, res) {
// 服务器分片校验函数，秒传及断点续传基础
//需后台给出对应的查询分片的接口进行分片文件验证
//这里是可以自定义的，但最好用官方的参数，如果后端判断文件存在，return一个skipUpload = true就行
let objMessage = JSON.parse(res);
if (objMessage.skipUpload) {
return true;
}
//如果当前分片比已经上传的最后一个分片要大，则允许上传
//如果小于等于，因为已经有序存在于服务器，为false不需要重传
return chunk.offset+1 > res.position;
},
simultaneousUploads: 1, //最大并发数，可以保证分片到达后端是有序的
},
}
},
1.2 Test multipart upload and instant transfer
- After understanding the above configuration, as long as the backend is written well, you can realize fragment upload and instant transmission, as shown in the figure:
Test first, if it is false, it means that there is no complete file, check the location of the fragment, start from position+1
Specifies the number of shards to start
- Then we upload the same file again, as shown in the figure:
For an existing file, send a trial request, if the backend returns skipUpload = true, it will be skipped and no more file upload requests will be sent
However, it should be noted that instant transmission can only be used for complete files , and it does not work for fragments. Breakpoint resume is for fragmentation , pay attention to the definition.
2. Backend
2.1 test interface and formal interface
Difference: we set test as the get interface, and it is a non-multipart file interface; while the official interface is used to receive files, corresponding to the two requests set by the front end above.
2.1.1 test interface
- Splicing urls to determine whether the complete file already exists
- If it exists, return a json string with skipUpload = true to the front end, skip the transmission, and realize the second transmission
- If it does not exist, return a json string with skipUpload = false and the latest fragment position = xx to the front end, and start the transmission
/**
* @Author Nineee
* @Date 2022/9/22 13:07
* @Description : 用于test快传 秒传 和 续传 的接口
* @param chunkNumber: 分片编号
* @param totalChunks: 总分片数
* @param totalSize: 总大小
* @param filename: 文件名
* @param curUrl: 当前位置
* @return Map
*/
@GetMapping("/uploadFile")
@ResponseBody
public Map uploadFile( @RequestParam("chunkNumber") String chunkNumber,
@RequestParam("totalChunks") String totalChunks,
@RequestParam("totalSize") String totalSize,
@RequestParam("filename") String filename,
@RequestParam("curUrl") String curUrl,
HttpServletRequest request, HttpServletResponse response) {
User user = (User)request.getAttribute("user");
int uid = user.getUid();
Map map = new HashMap();
boolean isTotalFileExist = Files.exists(Paths.get(store + uid + curUrl + "\\" + filename));
if(isTotalFileExist) {
//存在文件，秒传文件
map.put("skipUpload", true);
}else {
//未存在完整文件
map.put("skipUpload", false);
//应该检查目前到第几个分片，默认分片是有序的
String[] strs = filename.split("\\.");
String localUrl = store + uid + curUrl + "\\" + "tmp_"+strs[1]+"_"+strs[0]+"\\";
long count = fileService.findNewShard(localUrl);
map.put("position", count);
}
return map;
}
2.1.2 Formal interface
- Splice a temporary folder to store the fragments
- If the chunkNumber is not equal to totalChunks, it means that the transmission has not been completed (because our concurrency is 1, it must be transmitted in order), then pass false to the service, and directly IO writes the fragments into the temporary folder
- If the chunkNumber is equal to totalChunks, it means that this is the last fragment. After passing the last fragment to the temporary file, pass in true for the merge service
/**
* @Author Nineee
* @Date 2022/8/16 23:47
* @Description : 上传文件
* @param file: 需要上传的文件
* @param request: 请求体获取当前对象
* @return void
*/
@PostMapping("/uploadFile")
@ResponseBody
public void uploadFile( @RequestParam("file") MultipartFile file,
@RequestParam("chunkNumber") Integer chunkNumber,
@RequestParam("totalChunks") Integer totalChunks,
@RequestParam("totalSize") String totalSize,
@RequestParam("filename") String filename,
@RequestParam("curUrl") String curUrl,
HttpServletRequest request) {
User user = (User)request.getAttribute("user");
int uid = user.getUid();
//对于localUrl，如果不是末尾分片，我们应该加上一个tmp文件夹避免文件混乱。
//只有发起合并请求的时候再合并到源路径后删除tmp文件夹。
//注意，.需要转义
String[] strs = filename.split("\\.");
String localUrl = store + uid + curUrl + "\\" + "tmp_"+strs[1]+"_"+strs[0];
//不是最后一个分片，不需要合并
fileService.uploadFile(file, localUrl, ""+chunkNumber, false);
if(chunkNumber == totalChunks) {
//否则发起合并服务,merge合并
fileService.uploadFile(file, localUrl, filename, true);
}
}
2.2 Service layer
- The merge parameter indicates whether to merge
- If you do not need to merge, directly use the IO of the tool class to write to the temporary folder curUrl
- If you want to merge, use the utility class to merge
- After merging, delete the temporary folder recursively
@Override
/**
* @Author Nineee
* @Date 2022/8/16 23:07
* @Description : 上传文件
* @param file: multipart二进制文件流，也就是目标文件
* @param curUrl: 上传的目标地址
* @return Integer 1表示成功，0表示失败
*/
public Integer uploadFile(MultipartFile file, String localUrl, String filename, boolean merge) {
if(!merge) {
MultipartFileUtil.addFile(file, localUrl, filename);
}else {
MultipartFileUtil.mergeFileByRandomAccessFile(localUrl, filename);
//合并后删除tmp文件夹
try {
MultipartFileUtil.deleteDirByNio(localUrl);
} catch (Exception e) {
e.printStackTrace();
}
}
return 1;
}
/**
* @Author Nineee
* @Date 2022/9/22 16:41
* @Description : 找到最新的分片编号
* @param localUrl: 临时文件夹位置
* @return Long
*/
@Override
public Long findNewShard(String localUrl) {
File tempDir = new File(localUrl);
//如果连临时文件夹都没有，说明一定还没开始上传，不需要续传，直接0
if (!tempDir.exists()) {
tempDir.mkdirs();
return 0L;
}
//应该检查目前到第几个分片，默认分片是有序的
long count = 0;
try {
count = Files.list(Paths.get(localUrl)).count();
} catch (IOException e) {
e.printStackTrace();
}
return count;
}
2.3 Tool Introduction
2.3.1 addFile directly writes IO to fragments
- Because the set slice is 5M, which is relatively small, so using traditional BIO will not affect performance very much, it is relatively simple, and can be changed to NIO later
/**
* @Author Nineee
* @Date 2022/8/16 23:32
* @Description : 把multipartFile流文件写入到本地url中
* @param file: 文件流
* @param url: 本地路径
* @return void
*/
public static void addFile(MultipartFile file, String url, String filename){
OutputStream outputStream = null;
InputStream inputStream = null;
try {
inputStream = file.getInputStream();
//log.info("fileName="+fileName);
} catch (IOException e) {
e.printStackTrace();
}
try {
// 2、保存到临时文件
// 1K的数据缓冲
byte[] bs = new byte[1024];
// 读取到的数据长度
int len;
// 输出的文件流保存到本地文件
File tempFile = new File(url);
if (!tempFile.exists()) {
tempFile.mkdirs();
}
//跨平台写法，windows和linux都适用
outputStream = new FileOutputStream(tempFile.getPath()+"\\"+filename);
// 开始读取和写入os
while ((len = inputStream.read(bs)) != -1) {
outputStream.write(bs, 0, len);
}
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
} finally {
// 完毕，关闭所有链接
try {
outputStream.close();
inputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
2.3.2 mergeFileByRandomAccessFile merge files
- RandomAccessFile is a random access IO tool of NIO, which is a little faster than FileChannel of streaming IO
- Random access is to maintain two pointers, one head pointer, one limit pointer, and pass a File array into the loop to write to a single output url
/**
* @Author Nineee
* @Date 2022/9/22 13:53
* @Description : 通过随机访问IO即RandomAccessFile合并某文件夹里面的所有文件为一个文件
* @param fromUrl: 文件夹里装有所有的分片
* @param filename: 新的文件名
* @return void
*/
public static void mergeFileByRandomAccessFile(String fromUrl, String filename) {
String toUrl = fromUrl.substring(0, fromUrl.lastIndexOf("\\")+1)+"\\"+filename;
RandomAccessFile in = null;
RandomAccessFile out = null;
System.out.println(fromUrl);
File[] files = new File(fromUrl).listFiles();
//必须保证有序，名字根据编号排。避免默认的字典序，即1后面是10编号而不是2
Arrays.sort(files, (o1,o2) -> Integer.parseInt(o1.getName())-Integer.parseInt(o2.getName()));
try {
out = new RandomAccessFile(toUrl, "rw");
for (File file : files) {
in = new RandomAccessFile(file, "r");
int len = 0;
byte[] bt = new byte[BUF_SIZE];
while (-1 != (len = in.read(bt))) {
out.write(bt, 0, len);
}
in.close();
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (in != null) {
in.close();
}
if (out != null) {
out.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
2.3.3 deleteDirByNio delete folder
- If you use the delete of Files directly, there may be an exception that the file does not exist or the folder is not empty, so it needs to be deleted recursively
- The walkFileTree method in the Files class can obtain the file tree, which is convenient for recursive access. The file tree will follow the access order of files first and then folders
/**
* @Author Nineee
* @Date 2022/9/22 14:29
* @Description : 递归删除一个含有文件或者子文件夹的文件夹
* @param url: 文件夹地址
* @return void
*/
public static void deleteDirByNio(String url) throws Exception {
Path path = Paths.get(url);
Files.walkFileTree(path,
new SimpleFileVisitor<Path>() {
// 先去遍历删除文件
@Override
public FileVisitResult visitFile(Path file,
BasicFileAttributes attrs) throws IOException {
Files.delete(file);
return FileVisitResult.CONTINUE;
}
// 再去遍历删除目录
@Override
public FileVisitResult postVisitDirectory(Path dir,
IOException exc) throws IOException {
Files.delete(dir);
return FileVisitResult.CONTINUE;
}
}
);
}
2.4 Demonstration
2.4.1 Front-end selection folder
2.4.2 Temp folder
2.4.3 Sequential storage of file fragments
2.4.4 Merging and automatic deletion
3. Personally packaged uploader.vue
Achieved: fragmentation, instant transmission
<template>
<div>
<uploader
:options="this.options"
@file-added="this.fileAdded"
style="margin-right: 30px;float: left;">
<uploader-unsupport></uploader-unsupport>
<uploader-btn class="uploader-btn">
点击上传
</uploader-btn>
</uploader>
</div>
</template>
<script>
export default {
data() {
return {
options: {
target: "/file/uploadFile",
query: {
curUrl: this.$store.state.file.curUrl },
testMethod: "GET", //这里表示的是秒传的优先判断的请求方式
uploadMethod: "POST", //这里表示的是发送文件流的请求方式
method: "multipart",
fileParameterName: "file",
chunkSize: 5 * 1024 * 1024,
forceChunkSize: true, //强制每个分片都小于chunkSize
simultaneousUploads: 1, //最大并发数
testChunks: true,//是否开启服务器分片校验
checkChunkUploadedByResponse:
function (chunk, res) {
// 服务器分片校验函数，秒传及断点续传基础
//需后台给出对应的查询分片的接口进行分片文件验证
let objMessage = JSON.parse(res);//skipUpload、uploaded 需自己跟后台商量好参数名称
if (objMessage.skipUpload) {
return true;
}
return (objMessage.uploaded || []).indexOf(chunk.offset + 1) >= 0
},
maxChunkRetries: 2, //最大自动失败重试上传次数
},
}
},
methods:{
}
}
</script>
<style scoped>
</style>
4. Backend MutipartFileUtil
package com.nw.nwcloud.util;
import org.springframework.web.multipart.MultipartFile;
import java.io.*;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
public class MultipartFileUtil {
public static final int BUF_SIZE = 1024 * 1024;
/**
* @Author Nineee
* @Date 2022/8/16 23:32
* @Description : 把multipartFile流文件写入到本地url中
* @param file: 文件流
* @param url: 本地路径
* @return void
*/
public static void addFile(MultipartFile file, String url, String filename){
OutputStream outputStream = null;
InputStream inputStream = null;
try {
inputStream = file.getInputStream();
//log.info("fileName="+fileName);
} catch (IOException e) {
e.printStackTrace();
}
try {
// 2、保存到临时文件
// 1K的数据缓冲
byte[] bs = new byte[1024];
// 读取到的数据长度
int len;
// 输出的文件流保存到本地文件
File tempFile = new File(url);
if (!tempFile.exists()) {
tempFile.mkdirs();
}
//跨平台写法，windows和linux都适用
outputStream = new FileOutputStream(tempFile.getPath()+"\\"+filename);
// 开始读取和写入os
while ((len = inputStream.read(bs)) != -1) {
outputStream.write(bs, 0, len);
}
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
} finally {
// 完毕，关闭所有链接
try {
outputStream.close();
inputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
/**
* @Author Nineee
* @Date 2022/9/22 13:53
* @Description : 通过随机访问IO即RandomAccessFile合并某文件夹里面的所有文件为一个文件
* @param fromUrl: 文件夹里装有所有的分片
* @param filename: 新的文件名
* @return void
*/
public static void mergeFileByRandomAccessFile(String fromUrl, String filename) {
String toUrl = fromUrl.substring(0, fromUrl.lastIndexOf("\\")+1)+"\\"+filename;
RandomAccessFile in = null;
RandomAccessFile out = null;
System.out.println(fromUrl);
File[] files = new File(fromUrl).listFiles();
//必须保证有序，名字根据编号排。避免默认的字典序，即1后面是10编号而不是2
Arrays.sort(files, (o1,o2) -> Integer.parseInt(o1.getName())-Integer.parseInt(o2.getName()));
for(File f : files) {
System.out.println(f.getPath());
}
try {
out = new RandomAccessFile(toUrl, "rw");
for (File file : files) {
in = new RandomAccessFile(file, "r");
int len = 0;
byte[] bt = new byte[BUF_SIZE];
while (-1 != (len = in.read(bt))) {
out.write(bt, 0, len);
}
in.close();
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (in != null) {
in.close();
}
if (out != null) {
out.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
/**
* @Author Nineee
* @Date 2022/9/22 13:56
* @Description : 通过流式NIO即FileChannel合并某文件夹里面的所有文件为一个文件
* @param fromUrl: 文件夹里装有所有的分片
* @param filename: 新的文件名
* @return void
*/
public static void mergeFileByFileChannel(String fromUrl, String filename) {
String toUrl = fromUrl.substring(0, fromUrl.lastIndexOf("\\")+1)+"\\"+filename;
FileChannel outChannel = null;
FileChannel inChannel = null;
File[] files = new File(fromUrl).listFiles();
try {
outChannel = new FileOutputStream(toUrl).getChannel();
for (File file : files) {
inChannel = new FileInputStream(file).getChannel();
ByteBuffer bb = ByteBuffer.allocate(BUF_SIZE);
while (inChannel.read(bb) != -1) {
bb.flip();
outChannel.write(bb);
bb.clear();
}
inChannel.close();
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (outChannel != null) {
outChannel.close();
}
if (inChannel != null) {
inChannel.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
/**
* @Author Nineee
* @Date 2022/9/22 14:29
* @Description : 递归删除一个含有文件或者子文件夹的文件夹
* @param url: 文件夹地址
* @return void
*/
public static void deleteDirByNio(String url) throws Exception {
Path path = Paths.get(url);
Files.walkFileTree(path,
new SimpleFileVisitor<Path>() {
// 先去遍历删除文件
@Override
public FileVisitResult visitFile(Path file,
BasicFileAttributes attrs) throws IOException {
Files.delete(file);
return FileVisitResult.CONTINUE;
}
// 再去遍历删除目录
@Override
public FileVisitResult postVisitDirectory(Path dir,
IOException exc) throws IOException {
Files.delete(dir);
return FileVisitResult.CONTINUE;
}
}
);
}
}
5. Summary and preview
Implemented:
- Multipart upload
- Create a temporary folder to store shards
- Based on the concurrency number being 1, the slices are sorted, and when the last slice is encountered, it is the merge mark
- The fragmented files are merged in order through RandomAccessFile , because the fragmented files are named according to the number, and the numbers are ordered
- Implement recursive deletion of shard folders after merging
- The front end realizes the second transmission of complete files through the test request and the corresponding interface of the back end
- Achieved a single test request to obtain the current file fragment location, and realized the continuation of the upload
Not implemented:
-
Front-end concurrent upload
-
Shard verification
-
full file check