这两天发现一个视频网站有我想要的视频资源(https://m.jlszyy.org/play/2873-0-0.html)，想爬取，发现没那么简单。
里面是一个blob视频地址：
1 <video ... src="blob:https://jiexi.wysgtx.com/6764e0e3-2c88-44a4-bfea-341c32fb1697"></video>
不是直接一个mp4视频，没办法通过这个地址得到视频。
经过一天的琢磨，各种查资料，总算找到办法了。下面是分析和解决过程。
【分析】
F12进到调试模式，点击Network，选择XHR，然后强制刷新网页，看到底加载了啥：
重点关注前两个index.m3u8。前后两个m3u8的response内容分别是：
可见，第一个index.m3u8存储了第二个m3u8的地址，而第二个m3u8则存储了ts媒体分片的地址。
什么是m3u8，什么是ts媒体分片，参考[1]，这里不展开介绍。
目前只需要知道：将这些分片按顺序拼接到一起，就是我要的一个完整视频，这就够了。
最快的办法是，把每个视频的第二个m3u8（就是存储了所有ts分片地址的那个m3u8）地址复制出来，利用ffmpeg下载。
在CentOS环境下，利用下面这个命令成功下载到了视频。（参考[2]）以下载第23集为例：
1 ffmpeg -i https://sina.com-h-sina.com/20180906/18237_09096b18/800k/hls/index.m3u8 023.mp4
如果只是要下载视频，那么到此就结束了，文章下面也不用看了。（Windows平台有一个 M3U8 Downloader工具，也是基于ffmpeg实现的，用法更简单，参考[3]）
----------------------------- [ 华丽的分割线 ] -----------------------------
一开始我并没有正确使用这个ffmpeg（错用了第一个m3u8的地址），没下载到视频，所以放弃了这个好用的工具，转而自己撸代码去爬取......
从Headers可以获得m3u8的网址。
我的思路是，收集整部剧(36集)每个视频第一个m3u8的地址，然后解析并爬取第二个m3u8，接着就能爬取所有ts文件了。(36个url地址应该可以脚本爬取，不用手动收集，这是这个思路可优化的地方。当然，我没做这个优化。)
不废话，直接贴出可运行的python代码（最终版本：解析m3u8，多进程下载）：
1 from multiprocessing import Pool 2 import requests 3 import re 4 import os 5 6 ## 36个url 7 url_list=[ 8 'https://sina.com-h-sina.com/20180906/18215_c0fc8873/' , 9 'https://sina.com-h-sina.com/20180906/18216_80a7de01/' , 10 'https://sina.com-h-sina.com/20180906/18217_365c8b79/' , 11 'https://sina.com-h-sina.com/20180906/18218_f02026af/' , 12 'https://sina.com-h-sina.com/20180906/18219_7aca4012/' , 13 'https://sina.com-h-sina.com/20180906/18220_ed8429ba/' , 14 'https://cdn.youku-letv.net/20181130/11218_5aec4df0/' , 15 'https://cdn.youku-letv.net/20181130/11217_64a3d965/' , 16 'https://sina.com-h-sina.com/20180906/18223_5f07955a/' , 17 'https://sina.com-h-sina.com/20180906/18224_0f7d62f7/' , 18 'https://sina.com-h-sina.com/20180906/18225_70b7a36b/' , 19 'https://sina.com-h-sina.com/20180906/18226_e51fc698/' , 20 'https://sina.com-h-sina.com/20180906/18227_66271ec7/' , 21 'https://sina.com-h-sina.com/20180906/18228_026dee76/' , 22 'https://sina.com-h-sina.com/20180906/18229_371750e8/' , 23 'https://sina.com-h-sina.com/20180906/18230_997b5e5f/' , 24 'https://sina.com-h-sina.com/20180906/18231_88b8a296/' , 25 'https://sina.com-h-sina.com/20180906/18232_f20b2ca2/' , 26 'https://sina.com-h-sina.com/20180906/18233_3352578d/' , 27 'https://sina.com-h-sina.com/20180906/18234_347f6882/' , 28 'https://sina.com-h-sina.com/20180906/18235_44565aa7/' , 29 'https://cdn.youku-letv.net/20181130/11203_138f5930/' , 30 'https://sina.com-h-sina.com/20180906/18237_09096b18/' , 31 'https://sina.com-h-sina.com/20180906/18238_154e2c99/' , 32 'https://sina.com-h-sina.com/20180906/18239_e9fe736e/' , 33 'https://cdn.youku-letv.net/20181130/11199_aaf119a6/' , 34 'https://sina.com-h-sina.com/20180906/18241_e8d0a446/' , 35 'https://sina.com-h-sina.com/20180906/18242_ccfe1975/' , 36 'https://sina.com-h-sina.com/20180906/18243_316cafde/' , 37 'https://sina.com-h-sina.com/20180906/18244_c67bb56b/' , 38 'https://sina.com-h-sina.com/20180906/18245_f7707700/' , 39 'https://sina.com-h-sina.com/20180906/18246_01688d4e/' , 40 'https://sina.com-h-sina.com/20180906/18247_8bfc042c/' , 41 'https://sina.com-h-sina.com/20180906/18248_e85bfdc0/' , 42 'https://sina.com-h-sina.com/20180906/18249_b750a51f/' , 43 'https://cdn.youku-letv.net/20181130/11189_acdfba0e/' 44 ] 45 46 ## [注意]正阳门下的视频的第一个m3u8是指向另一个m3u8，并不是直接指向ts媒体分片 47 index='index.m3u8' 48 savefile_path='E://Downloads//ZhengYangMenXia//' 49 50 51 52 def download_task(n): 53 i = int(n) 54 print('task %d start!' % i) 55 # os.mkdir(savefile_path+str(i)+'//') ## 创建文件夹。如果手动创建了文件夹，那就注释掉这一句 56 data=requests.get(url_list[i]+index) 57 58 lines=str.split(data.text, '\n') ## 字符串处理。从第一层m3u8里面取得真正的m3u8地址 59 m3u8=lines[2] 60 pattern="index.m3u8" 61 m3u8_directory=re.sub(pattern,"", m3u8) ## 取文件夹路径，后面会用到 62 63 data = requests.get(url_list[i] + m3u8) 64 ''' 65 # 保存m3u8文件。此处注释掉，m3u8的内容就不存了。 66 file=open(savefile_path+str(i)+'//'+index, 'w') 67 file.write(data.text) 68 file.close() 69 ''' 70 lines=str.split(data.text,'\n') 71 pattern=r".*ts" 72 # 保存ts文件(注意，是二进制的) 73 video = open(savefile_path + '//' + '正阳门下-' + str(i).zfill(3) + '.mp4', 'wb') 74 for j in range(0,len(lines)): 75 if re.match(pattern,lines[j]): 76 print("Task %d Downloading %s..." % (i, lines[j])) 77 data = requests.get(url_list[i]+m3u8_directory+lines[j]) 78 video.write(data.content) 79 video.close() 80 print('Task %d done!' % i) 81 82 83 84 if __name__=='__main__': ## 多进程程序，一定要有main函数。 85 pool=Pool(4) ## 4进程下载 86 for i in range(0,len(url_list)): 87 pool.apply_async(download_task, args=(i,)) 88 print('Task %d has been submited' % i) 89 print('Waiting for all subprocesses done...') 90 pool.close() 91 pool.join() 92 print('All subprocesses done.')
这份代码是这么写成的：
【1】收集数据，体现为一个url列表：
1 ## 36个url 2 url_list=[ 3 'https://sina.com-h-sina.com/20180906/18215_c0fc8873/' , 4 'https://sina.com-h-sina.com/20180906/18216_80a7de01/' , 5 'https://sina.com-h-sina.com/20180906/18217_365c8b79/' , 6 'https://sina.com-h-sina.com/20180906/18218_f02026af/' , 7 'https://sina.com-h-sina.com/20180906/18219_7aca4012/' , 8 'https://sina.com-h-sina.com/20180906/18220_ed8429ba/' , 9 'https://cdn.youku-letv.net/20181130/11218_5aec4df0/' , 10 'https://cdn.youku-letv.net/20181130/11217_64a3d965/' , 11 'https://sina.com-h-sina.com/20180906/18223_5f07955a/' , 12 'https://sina.com-h-sina.com/20180906/18224_0f7d62f7/' , 13 'https://sina.com-h-sina.com/20180906/18225_70b7a36b/' , 14 'https://sina.com-h-sina.com/20180906/18226_e51fc698/' , 15 'https://sina.com-h-sina.com/20180906/18227_66271ec7/' , 16 'https://sina.com-h-sina.com/20180906/18228_026dee76/' , 17 'https://sina.com-h-sina.com/20180906/18229_371750e8/' , 18 'https://sina.com-h-sina.com/20180906/18230_997b5e5f/' , 19 'https://sina.com-h-sina.com/20180906/18231_88b8a296/' , 20 'https://sina.com-h-sina.com/20180906/18232_f20b2ca2/' , 21 'https://sina.com-h-sina.com/20180906/18233_3352578d/' , 22 'https://sina.com-h-sina.com/20180906/18234_347f6882/' , 23 'https://sina.com-h-sina.com/20180906/18235_44565aa7/' , 24 'https://cdn.youku-letv.net/20181130/11203_138f5930/' , 25 'https://sina.com-h-sina.com/20180906/18237_09096b18/' , 26 'https://sina.com-h-sina.com/20180906/18238_154e2c99/' , 27 'https://sina.com-h-sina.com/20180906/18239_e9fe736e/' , 28 'https://cdn.youku-letv.net/20181130/11199_aaf119a6/' , 29 'https://sina.com-h-sina.com/20180906/18241_e8d0a446/' , 30 'https://sina.com-h-sina.com/20180906/18242_ccfe1975/' , 31 'https://sina.com-h-sina.com/20180906/18243_316cafde/' , 32 'https://sina.com-h-sina.com/20180906/18244_c67bb56b/' , 33 'https://sina.com-h-sina.com/20180906/18245_f7707700/' , 34 'https://sina.com-h-sina.com/20180906/18246_01688d4e/' , 35 'https://sina.com-h-sina.com/20180906/18247_8bfc042c/' , 36 'https://sina.com-h-sina.com/20180906/18248_e85bfdc0/' , 37 'https://sina.com-h-sina.com/20180906/18249_b750a51f/' , 38 'https://cdn.youku-letv.net/20181130/11189_acdfba0e/' 39 ]
【2】先试验性地将一集下载下来（也就是这一步确认了：只要将ts文件下载下来拼接到一起，就能得到一个完整视频）
1 # coding=utf8 2 import requests 3 4 url='https://sina.com-h-sina.com/20180906/18215_c0fc8873/800k/hls/f997a136b73' 5 6 for i in range(0,675): 7 f = open("E:/Downloads/tmp/%03d.ts" % i, 'wb') 8 data = requests.get(url + str(i).zfill(3) + '.ts') 9 f.write(data.content) 10 f.close() 11 print('ts %03d OK' % i)
【3】遍历列表，爬取每个视频的第一个m3u8，然后解析出第二个m3u8的地址，接着爬取第二个m3u8，最后解析并爬取所有ts。
这也是我写的第一个版本
1 import requests 2 import re 3 import os 4 5 url_list=[ 6 'https://sina.com-h-sina.com/20180906/18215_c0fc8873/' , 7 'https://sina.com-h-sina.com/20180906/18216_80a7de01/' , 8 'https://sina.com-h-sina.com/20180906/18217_365c8b79/' , 9 'https://sina.com-h-sina.com/20180906/18218_f02026af/' , 10 'https://sina.com-h-sina.com/20180906/18219_7aca4012/' , 11 'https://sina.com-h-sina.com/20180906/18220_ed8429ba/' , 12 'https://cdn.youku-letv.net/20181130/11218_5aec4df0/' , 13 'https://cdn.youku-letv.net/20181130/11217_64a3d965/' , 14 'https://sina.com-h-sina.com/20180906/18223_5f07955a/' , 15 'https://sina.com-h-sina.com/20180906/18224_0f7d62f7/' , 16 'https://sina.com-h-sina.com/20180906/18225_70b7a36b/' , 17 'https://sina.com-h-sina.com/20180906/18226_e51fc698/' , 18 'https://sina.com-h-sina.com/20180906/18227_66271ec7/' , 19 'https://sina.com-h-sina.com/20180906/18228_026dee76/' , 20 'https://sina.com-h-sina.com/20180906/18229_371750e8/' , 21 'https://sina.com-h-sina.com/20180906/18230_997b5e5f/' , 22 'https://sina.com-h-sina.com/20180906/18231_88b8a296/' , 23 'https://sina.com-h-sina.com/20180906/18232_f20b2ca2/' , 24 'https://sina.com-h-sina.com/20180906/18233_3352578d/' , 25 'https://sina.com-h-sina.com/20180906/18234_347f6882/' , 26 'https://sina.com-h-sina.com/20180906/18235_44565aa7/' , 27 'https://cdn.youku-letv.net/20181130/11203_138f5930/' , 28 'https://sina.com-h-sina.com/20180906/18237_09096b18/' , 29 'https://sina.com-h-sina.com/20180906/18238_154e2c99/' , 30 'https://sina.com-h-sina.com/20180906/18239_e9fe736e/' , 31 'https://cdn.youku-letv.net/20181130/11199_aaf119a6/' , 32 'https://sina.com-h-sina.com/20180906/18241_e8d0a446/' , 33 'https://sina.com-h-sina.com/20180906/18242_ccfe1975/' , 34 'https://sina.com-h-sina.com/20180906/18243_316cafde/' , 35 'https://sina.com-h-sina.com/20180906/18244_c67bb56b/' , 36 'https://sina.com-h-sina.com/20180906/18245_f7707700/' , 37 'https://sina.com-h-sina.com/20180906/18246_01688d4e/' , 38 'https://sina.com-h-sina.com/20180906/18247_8bfc042c/' , 39 'https://sina.com-h-sina.com/20180906/18248_e85bfdc0/' , 40 'https://sina.com-h-sina.com/20180906/18249_b750a51f/' , 41 'https://cdn.youku-letv.net/20181130/11189_acdfba0e/' 42 ] 43 ## [注意]正阳门下的视频的第一个m3u8是指向另一个m3u8，并不是直接指向ts媒体分片 44 index='index.m3u8' 45 savefile_path='E://Downloads//ZhengYangMenXia//' 46 47 for i in range(0,len(url_list)): 48 # os.mkdir(savefile_path+str(i)+'//') ## 创建文件夹。如果手动创建了文件夹，那就注释掉这一句 49 data=requests.get(url_list[i]+index) 50 51 lines=str.split(data.text, '\n') ## 字符串处理。从第一层m3u8里面取得真正的m3u8地址 52 m3u8=lines[2] 53 pattern="index.m3u8" 54 m3u8_directory=re.sub(pattern,"", m3u8) ## 取文件夹路径，后面会用到 55 56 data = requests.get(url_list[i] + m3u8) 57 ''' 58 # 保存m3u8文件。注释掉，就不存了。 59 file=open(savefile_path+str(i)+'//'+index, 'w') 60 file.write(data.text) 61 file.close() 62 ''' 63 lines=str.split(data.text,'\n') 64 pattern=r".*ts" 65 # 保存ts文件(注意，是二进制的) 66 video = open(savefile_path + str(i) + '//' + '正阳门下-' + str(i).zfill(3) + '.mp4', 'wb') 67 for j in range(0,len(lines)): 68 if re.match(pattern,lines[j]): 69 print("Downloading %s..." % lines[j]) 70 data = requests.get(url_list[i]+m3u8_directory+lines[j]) 71 video.write(data.content) 72 video.close()
【4】单进程下载不够快，改成多进程吧。于是有了上面贴出来的第一份可运行代码。（参考[5]）
【参考】
[1] m3u8的维基百科介绍：https://wikipedia.hk.wjbk.site/baike-M3U
[2] Linux平台，通过 ffmpeg命令下载视频：https://dzt666.cn/index.php/archives/203/
[3] Windows平台，通过 M3U8 Downloader 下载：https://blog.csdn.net/yjclsx/article/details/88948860
[4] m3u8和ts媒体分片的介绍：https://www.jianshu.com/p/4f6db32fc76a
[5] python多进程：https://www.liaoxuefeng.com/wiki/1016959663602400/1017628290184064
[6] 带来一点启发的参考资料：https://blog.csdn.net/angry_mills/article/details/82705595