爬虫如何爬取某领域下百家号作者列表

其他 2019-04-19 12:13:13 阅读次数: 0
版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/ucsheep/article/details/89285420

分析百家号的作者的数据信息，发现在百家号中，每一个 appid 确定一个作者。（appid，听着也和作者author毫无联系，随即也想到，百家号早就支持作者通过API调用的形式去发布视频或者图文内容了，看来百度也是早有安排了。）

爬虫想要尽可能多地获得某个领域下的作者列表，而百度并不提供这样的接口，所以只能另辟蹊径。

常规的做法分两种：

  • 1 去在某领域下一直爬文章，顺便获取用户信息，做好去重，等着数据增长。
  • 2 由于1的方法实在太守株待兔了点，效率也低，所以第二种就是我们异想天开地：能不能通过搜索引擎直接拿到？

第一种，可行，但是需要考虑其推荐算法的影响，操作不当在后期会有巨量的请求返回重复的数据，越往后效率越低。
所以，我们着重讲第二种

本来也只是我们异想天开，但是事情往往就是那么凑巧，突然发现百度的搜索引擎还真有这样的一个口子。一般我们在PC端搜索关键词 [百家号+军事] ，会发现没有我们期望的结果出现。如下图
PC端搜索
但是假如我们伪装成移动端搜索呢？简直是惊喜！
在这里插入图片描述
看到没，更多百家号！点击进入，一直向下滑动，你会源源不断地获得新的百家号作者！

每一页是类似下面这样子的请求

https://www.baidu.com/sf?
word=%E7%99%BE%E5%AE%B6%E5%8F%B7%2B%E5%86%9B%E4%BA%8B
&pd=cambrian_list
&atn=index
&title=%E7%99%BE%E5%AE%B6%E5%8F%B7%2B%E5%86%9B%E4%BA%8B
&lid=9080249029523443283
&ms=1&frsrcid=206&frorder=1&pn=20&data_type=json

其中，word和title解析后都是
百家号%2B军事

接着，我们分析连续的10个请求如下

序号 请求
1 https://www.baidu.com/sf?word=百家号%2B军事&pd=cambrian_list&atn=index&title=百家号%2B军事&lid=9080249029523443283&ms=1&frsrcid=206&frorder=1&pn=20&data_type=json
2 https://www.baidu.com/sf?word=百家号%2B军事&pd=cambrian_list&atn=index&title=百家号%2B军事&lid=9080249029523443283&ms=1&frsrcid=206&frorder=1&pn=30&data_type=json
3 https://www.baidu.com/sf?word=百家号%2B军事&pd=cambrian_list&atn=index&title=百家号%2B军事&lid=9080249029523443283&ms=1&frsrcid=206&frorder=1&pn=40&data_type=json
4 https://www.baidu.com/sf?word=百家号%2B军事&pd=cambrian_list&atn=index&title=百家号%2B军事&lid=9080249029523443283&ms=1&frsrcid=206&frorder=1&pn=50&data_type=json
5 https://www.baidu.com/sf?word=百家号%2B军事&pd=cambrian_list&atn=index&title=百家号%2B军事&lid=9080249029523443283&ms=1&frsrcid=206&frorder=1&pn=60&data_type=json
6 https://www.baidu.com/sf?word=百家号%2B军事&pd=cambrian_list&atn=index&title=百家号%2B军事&lid=9080249029523443283&ms=1&frsrcid=206&frorder=1&pn=70&data_type=json
7 https://www.baidu.com/sf?word=百家号%2B军事&pd=cambrian_list&atn=index&title=百家号%2B军事&lid=9080249029523443283&ms=1&frsrcid=206&frorder=1&pn=80&data_type=json
8 https://www.baidu.com/sf?word=百家号%2B军事&pd=cambrian_list&atn=index&title=百家号%2B军事&lid=9080249029523443283&ms=1&frsrcid=206&frorder=1&pn=90&data_type=json
9 https://www.baidu.com/sf?word=百家号%2B军事&pd=cambrian_list&atn=index&title=百家号%2B军事&lid=9080249029523443283&ms=1&frsrcid=206&frorder=1&pn=100&data_type=json
10 https://www.baidu.com/sf?word=百家号%2B军事&pd=cambrian_list&atn=index&title=百家号%2B军事&lid=9080249029523443283&ms=1&frsrcid=206&frorder=1&pn=110&data_type=json

规律很明显了

pn字段，随着页数以10递增，它应该就是百度用于分页的游标，即排序后从pn开始，连取10条，返回。

为了验证以上推断的正确性，我们令pn=5000000,发动一起请求，请求结果如下(已略去其余信息)

"list": [{
			"batchid": 0,
			"logo": "https:\/\/cambrian-images.cdn.bcebos.com\/60208daa9248a23cf3952959d78fbdfa_1553499630421535.jpeg@w_100,h_100",
			"title": "百家军事网",
			"des": "军事访谈日理万机，国家志强",
			"url": "https:\/\/author.baidu.com\/home\/1553499630421535?from=dusite_sresults",
			"appid": "1553499630421535",
			"platform": 2,
			"v_type": 0,
			"tcUrl": "http:\/\/m.baidu.com\/from=0\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm@0,sz@224_220,ta@qbase___24_70.0\/baiduid=C525C3CCAD9F56EF4AB522E6D4B3C18A\/w=760_10_\/t=iphone\/l=1\/tc?ref=www_iphone&lid=9080249029523443283&isAtom=1&is_baidu=0&clk_info={"applid":"7530378237642659127","apptpl":"normal","frsrcid":"206","frorder":"1"}&module=sf&wd=&eqid=6881471c0705d400100000035cb1a520&w_qd=IlPT2AEptyoA_ykytR9c-PKxIFGQt9RNok6q&bdver=1&tcplug=1&sec=37785&di=32ba00383f0ba8c4&bdenc=1&nsrc=IlPT2AEptyoA_yixCFOxCGZb8c3JV3T5ABOIQSZLAHSwnESzbbrgHtkfEFX8L8KTZpPPdTPLshk1uX_c07Io7BV1q0dnpTthlCDwdsigxdLXWM2yv11631P3XD2e-LGf7Kwfd2B8RNU8",
			"isSub": false,
			"isLoad": false
		}, {
			"batchid": 0,
			"logo": "https:\/\/cambrian-images.cdn.bcebos.com\/02cd688146893af2f958476052183a2a_1570407697554051.jpeg@w_100,h_100",
			"title": "军事小频道",
			"des": "每天更新不同的文章,让你开心一整天~",
			"url": "https:\/\/author.baidu.com\/home\/1570407697554051?from=dusite_sresults",
			"appid": "1570407697554051",
			"platform": 2,
			"v_type": 0,
			"tcUrl": "http:\/\/m.baidu.com\/from=0\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm@0,sz@224_220,ta@qbase___24_70.0\/baiduid=C525C3CCAD9F56EF4AB522E6D4B3C18A\/w=760_10_\/t=iphone\/l=1\/tc?ref=www_iphone&lid=9080249029523443283&isAtom=1&is_baidu=0&clk_info={"applid":"7530378237642659127","apptpl":"normal","frsrcid":"206","frorder":"1"}&module=sf&wd=&eqid=6881471c0705d400100000035cb1a520&w_qd=IlPT2AEptyoA_ykytR9c-PKxIFGQt9RNok6q&bdver=1&tcplug=1&sec=37785&di=e840d88f39c9cade&bdenc=1&nsrc=IlPT2AEptyoA_yixCFOxCGZb8c3JV3T5ABOIQSZLAHSwnESzbbrgHtkfEFX8L8KTZpPPdTPNsRkFx8_i_GEj8RJ3r0dnpTthlCDwdsigxdLXWM2yv11631P3XD2e-LGf7Kwfd2B8RNU8",
			"isSub": false,
			"isLoad": false
		}, {
			"batchid": 0,
			"logo": "https:\/\/cambrian-images.cdn.bcebos.com\/cbf762f31adbac3c97ee4c19338cc45d_1555602785265779.jpeg@w_100,h_100",
			"title": "百家军事",
			"des": "最新全球军事动态，武器装备尽在百家军事。",
			"url": "https:\/\/author.baidu.com\/home\/1555602785265779?from=dusite_sresults",
			"appid": "1555602785265779",
			"platform": 2,
			"v_type": 0,
			"tcUrl": "http:\/\/m.baidu.com\/from=0\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm@0,sz@224_220,ta@qbase___24_70.0\/baiduid=C525C3CCAD9F56EF4AB522E6D4B3C18A\/w=760_10_\/t=iphone\/l=1\/tc?ref=www_iphone&lid=9080249029523443283&isAtom=1&is_baidu=0&clk_info={"applid":"7530378237642659127","apptpl":"normal","frsrcid":"206","frorder":"1"}&module=sf&wd=&eqid=6881471c0705d400100000035cb1a520&w_qd=IlPT2AEptyoA_ykytR9c-PKxIFGQt9RNok6q&bdver=1&tcplug=1&sec=37785&di=4f0f37910dbff044&bdenc=1&nsrc=IlPT2AEptyoA_yixCFOxCGZb8c3JV3T5ABOIQSZLAHSwnESzbbrgHtkfEFX8L8KTZpPPdTPLtBcFwn3j_mQk8BNAt0dnpTthlCDwdsigxdLXWM2yv11631P3XD2e-LGf7Kwfd2B8RNU8",
			"isSub": false,
			"isLoad": false
		}, {
			"batchid": 0,
			"logo": "https:\/\/cambrian-images.cdn.bcebos.com\/75e329a8747f7689366ef7b46c82f4ee_1570624429637509.jpeg@w_100,h_100",
			"title": "军事家",
			"des": "军事家，以军事为家，立足中华关注全球军情",
			"url": "https:\/\/author.baidu.com\/home\/1570624429637509?from=dusite_sresults",
			"appid": "1570624429637509",
			"platform": 2,
			"v_type": 3,
			"tcUrl": "http:\/\/m.baidu.com\/from=0\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm@0,sz@224_220,ta@qbase___24_70.0\/baiduid=C525C3CCAD9F56EF4AB522E6D4B3C18A\/w=760_10_\/t=iphone\/l=1\/tc?ref=www_iphone&lid=9080249029523443283&isAtom=1&is_baidu=0&clk_info={"applid":"7530378237642659127","apptpl":"normal","frsrcid":"206","frorder":"1"}&module=sf&wd=&eqid=6881471c0705d400100000035cb1a520&w_qd=IlPT2AEptyoA_ykytR9c-PKxIFGQt9RNok6q&bdver=1&tcplug=1&sec=37785&di=f1d5ee1f3b276c8e&bdenc=1&nsrc=IlPT2AEptyoA_yixCFOxCGZb8c3JV3T5ABOIQSZLAHSwnESzbbrgHtkfEFX8L8KTZpPPdTPNsRcDxHCd2mAp8hV0t0dnpTthlCDwdsigxdLXWM2yv11631P3XD2e-LGf7Kwfd2B8RNU8",
			"isSub": false,
			"isLoad": false,
			"vTypeUrl": "\/\/b.bdstatic.com\/searchbox\/icms\/searchbox\/img\/vip-3.png"
		}, {
			"batchid": 0,
			"logo": "https:\/\/cambrian-images.cdn.bcebos.com\/d4743a0cda2d319c68512bfa43c2dd07_1572768436626568.jpeg@w_100,h_100",
			"title": "央视军事",
			"des": "中央电视台军事节目中心官方发布平台",
			"url": "https:\/\/author.baidu.com\/home\/1572768436626568?from=dusite_sresults",
			"appid": "1572768436626568",
			"platform": 3,
			"v_type": 2,
			"tcUrl": "http:\/\/m.baidu.com\/from=0\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm@0,sz@224_220,ta@qbase___24_70.0\/baiduid=C525C3CCAD9F56EF4AB522E6D4B3C18A\/w=760_10_\/t=iphone\/l=1\/tc?ref=www_iphone&lid=9080249029523443283&isAtom=1&is_baidu=0&clk_info={"applid":"7530378237642659127","apptpl":"normal","frsrcid":"206","frorder":"1"}&module=sf&wd=&eqid=6881471c0705d400100000035cb1a520&w_qd=IlPT2AEptyoA_ykytR9c-PKxIFGQt9RNok6q&bdver=1&tcplug=1&sec=37785&di=5b3b280e0044b2bf&bdenc=1&nsrc=IlPT2AEptyoA_yixCFOxCGZb8c3JV3T5ABOIQSZLAHSwnESzbbrgHtkfEFX8L8KTZpPPdTPNsxgHuHCc_WAo8xVBtLdnpTthlCDwdsigxdLXWM2yv11631P3XD2e-LGf7Kwfd2B8RNU8",
			"isSub": false,
			"isLoad": false,
			"vTypeUrl": "\/\/b.bdstatic.com\/searchbox\/icms\/searchbox\/img\/vip-2.png"
		}, {
			"batchid": 0,
			"logo": "https:\/\/cambrian-images.cdn.bcebos.com\/ab6f1f29a74d5a2ff3759cac852ce463_1563044131013992.jpeg@w_100,h_100",
			"title": "百家中国军事",
			"des": "富国强兵，居安思危。",
			"url": "https:\/\/author.baidu.com\/home\/1563044131013992?from=dusite_sresults",
			"appid": "1563044131013992",
			"platform": 2,
			"v_type": 0,
			"tcUrl": "http:\/\/m.baidu.com\/from=0\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm@0,sz@224_220,ta@qbase___24_70.0\/baiduid=C525C3CCAD9F56EF4AB522E6D4B3C18A\/w=760_10_\/t=iphone\/l=1\/tc?ref=www_iphone&lid=9080249029523443283&isAtom=1&is_baidu=0&clk_info={"applid":"7530378237642659127","apptpl":"normal","frsrcid":"206","frorder":"1"}&module=sf&wd=&eqid=6881471c0705d400100000035cb1a520&w_qd=IlPT2AEptyoA_ykytR9c-PKxIFGQt9RNok6q&bdver=1&tcplug=1&sec=37785&di=650295acfdd71719&bdenc=1&nsrc=IlPT2AEptyoA_yixCFOxCGZb8c3JV3T5ABOIQSZLAHSwnESzbbrgHtkfEFX8L8KTZpPPdTPMsh5JxHWc0mYn7hlCrrdnpTthlCDwdsigxdLXWM2yv11631P3XD2e-LGf7Kwfd2B8RNU8",
			"isSub": false,
			"isLoad": false
		}, {
			"batchid": 0,
			"logo": "https:\/\/cambrian-images.cdn.bcebos.com\/7aa8393aad3aa3f93b1c0278eaddb72f_1562683489652294.jpeg@w_100,h_100",
			"title": "百家第一军事",
			"des": "军事无限激情，百家第一军事让你享受",
			"url": "https:\/\/author.baidu.com\/home\/1562683489652294?from=dusite_sresults",
			"appid": "1562683489652294",
			"platform": 2,
			"v_type": 0,
			"tcUrl": "http:\/\/m.baidu.com\/from=0\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm@0,sz@224_220,ta@qbase___24_70.0\/baiduid=C525C3CCAD9F56EF4AB522E6D4B3C18A\/w=760_10_\/t=iphone\/l=1\/tc?ref=www_iphone&lid=9080249029523443283&isAtom=1&is_baidu=0&clk_info={"applid":"7530378237642659127","apptpl":"normal","frsrcid":"206","frorder":"1"}&module=sf&wd=&eqid=6881471c0705d400100000035cb1a520&w_qd=IlPT2AEptyoA_ykytR9c-PKxIFGQt9RNok6q&bdver=1&tcplug=1&sec=37785&di=273a227495d17bac&bdenc=1&nsrc=IlPT2AEptyoA_yixCFOxCGZb8c3JV3T5ABOIQSZLAHSwnESzbbrgHtkfEFX8L8KTZpPPdTPMsxc2w8Cj2mAj7xBCqLdnpTthlCDwdsigxdLXWM2yv11631P3XD2e-LGf7Kwfd2B8RNU8",
			"isSub": false,
			"isLoad": false
		}, {
			"batchid": 0,
			"logo": "https:\/\/cambrian-images.cdn.bcebos.com\/ef5b910ea28be64b7893c8b72e39628d_1569984336032587.jpeg@w_100,h_100",
			"title": "军事帝",
			"des": "军事帝，提供原创武器视频解说！",
			"url": "https:\/\/author.baidu.com\/home\/1569984336032587?from=dusite_sresults",
			"appid": "1569984336032587",
			"platform": 3,
			"v_type": 2,
			"tcUrl": "http:\/\/m.baidu.com\/from=0\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm@0,sz@224_220,ta@qbase___24_70.0\/baiduid=C525C3CCAD9F56EF4AB522E6D4B3C18A\/w=760_10_\/t=iphone\/l=1\/tc?ref=www_iphone&lid=9080249029523443283&isAtom=1&is_baidu=0&clk_info={"applid":"7530378237642659127","apptpl":"normal","frsrcid":"206","frorder":"1"}&module=sf&wd=&eqid=6881471c0705d400100000035cb1a520&w_qd=IlPT2AEptyoA_ykytR9c-PKxIFGQt9RNok6q&bdver=1&tcplug=1&sec=37785&di=57eea283e97f27b9&bdenc=1&nsrc=IlPT2AEptyoA_yixCFOxCGZb8c3JV3T5ABOIQSZLAHSwnESzbbrgHtkfEFX8L8KTZpPPdTPMqBY2xHOc_WYp7xVDqbdnpTthlCDwdsigxdLXWM2yv11631P3XD2e-LGf7Kwfd2B8RNU8",
			"isSub": false,
			"isLoad": false,
			"vTypeUrl": "\/\/b.bdstatic.com\/searchbox\/icms\/searchbox\/img\/vip-2.png"
		}, {
			"batchid": 0,
			"logo": "https:\/\/cambrian-images.cdn.bcebos.com\/25ff7a88e736c595816b6ea558545513_1563931682954648.jpeg@w_100,h_100",
			"title": "军事百家号",
			"des": "速度就造就了成功，没有速度就没有成功。",
			"url": "https:\/\/author.baidu.com\/home\/1563931682954648?from=dusite_sresults",
			"appid": "1563931682954648",
			"platform": 2,
			"v_type": 0,
			"tcUrl": "http:\/\/m.baidu.com\/from=0\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm@0,sz@224_220,ta@qbase___24_70.0\/baiduid=C525C3CCAD9F56EF4AB522E6D4B3C18A\/w=760_10_\/t=iphone\/l=1\/tc?ref=www_iphone&lid=9080249029523443283&isAtom=1&is_baidu=0&clk_info={"applid":"7530378237642659127","apptpl":"normal","frsrcid":"206","frorder":"1"}&module=sf&wd=&eqid=6881471c0705d400100000035cb1a520&w_qd=IlPT2AEptyoA_ykytR9c-PKxIFGQt9RNok6q&bdver=1&tcplug=1&sec=37785&di=7ed6ff8579046d07&bdenc=1&nsrc=IlPT2AEptyoA_yixCFOxCGZb8c3JV3T5ABOIQSZLAHSwnESzbbrgHtkfEFX8L8KTZpPPdTPMshYCwX_j0W5j8RR4tLdnpTthlCDwdsigxdLXWM2yv11631P3XD2e-LGf7Kwfd2B8RNU8",
			"isSub": false,
			"isLoad": false
		}, {
			"batchid": 0,
			"logo": "https:\/\/cambrian-images.cdn.bcebos.com\/5bffec62581df1356a27a88bcd540120_1554754061100166.jpeg@w_100,h_100",
			"title": "军事头条网",
			"des": "汇聚世界各国的精彩军事内容",
			"url": "https:\/\/author.baidu.com\/home\/1554754061100166?from=dusite_sresults",
			"appid": "1554754061100166",
			"platform": 2,
			"v_type": 0,
			"tcUrl": "http:\/\/m.baidu.com\/from=0\/bd_page_type=1\/ssid=0\/uid=0\/pu=usm@0,sz@224_220,ta@qbase___24_70.0\/baiduid=C525C3CCAD9F56EF4AB522E6D4B3C18A\/w=760_10_\/t=iphone\/l=1\/tc?ref=www_iphone&lid=9080249029523443283&isAtom=1&is_baidu=0&clk_info={"applid":"7530378237642659127","apptpl":"normal","frsrcid":"206","frorder":"1"}&module=sf&wd=&eqid=6881471c0705d400100000035cb1a520&w_qd=IlPT2AEptyoA_ykytR9c-PKxIFGQt9RNok6q&bdver=1&tcplug=1&sec=37785&di=48a097be1fddad4a&bdenc=1&nsrc=IlPT2AEptyoA_yixCFOxCGZb8c3JV3T5ABOIQSZLAHSwnESzbbrgHtkfEFX8L8KTZpPPdTPLtRgIxHSh0mUm7RFBqrdnpTthlCDwdsigxdLXWM2yv11631P3XD2e-LGf7Kwfd2B8RNU8",
			"isSub": false,
			"isLoad": false
		}],

当pn远超总体数量时，百度返回的是第一页的数据(如图2)。
这样，我们还可以利用这一特性，使用2分法快速地算出当前关键词的百家号作者的总体数量。方便我们确定爬虫爬取的任务规模。

最后

但是，上面的方法也是有弊端的，爬到的百家号作者都是作者名包含关键词的，这样，我们要获取某个领域的作者，就需要首先构造出该领域下的若干关键词。当然，也有可能拿到的作者名虽然包含该关键词，当时并不是该领域的作者。

猜你喜欢

转载自blog.csdn.net/ucsheep/article/details/89285420
今日推荐
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
中国码农的“35岁魔咒”
蘭雅 CorelDRAW 插件 2024.5.1 国际劳动节版，免费下载
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
周排行
基本数据类型封装类比较 Java源码解读(一) 8种基本类型对应的封装类型
JS实现无缝滚动上
深入解析HashMap原理（基于JDK1.8）
mysql的连接池
关于.htc
linux下的ubuntu12.04图形界面
【数论】好推不好记的扩展欧几里德
设备树详解
cscope + tags 简单设置
xml学习
每日归档
更多
2024-05-09(35)
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022