Python爬虫系列之----Scrapy里settings.py配置文件详解

其他 2018-07-21 06:41:41 阅读次数: 0

让我们先来看下它里面的内容:


  1. # -*- coding: utf-8 -*-
  3. # Scrapy settings for demo1 project
  4. #
  5. # For simplicity, this file contains only settings considered important or
  6. # commonly used. You can find more settings consulting the documentation:
  7. #
  8. # http://doc.scrapy.org/en/latest/topics/settings.html
  9. # http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
  10. # http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
  12. BOT_NAME = 'demo1' #Scrapy项目的名字,这将用来构造默认 User-Agent,同时也用来log,当您使用 startproject 命令创建项目时其也被自动赋值。
  14. SPIDER_MODULES = ['demo1.spiders'] #Scrapy搜索spider的模块列表 默认: [xxx.spiders]
  15. NEWSPIDER_MODULE = 'demo1.spiders' #使用 genspider 命令创建新spider的模块。默认: 'xxx.spiders'
  18. #爬取的默认User-Agent，除非被覆盖
  19. #USER_AGENT = 'demo1 (+http://www.yourdomain.com)'
  21. #如果启用,Scrapy将会采用 robots.txt策略
  22. ROBOTSTXT_OBEY = True
  24. #Scrapy downloader 并发请求(concurrent requests)的最大值,默认: 16
  25. #CONCURRENT_REQUESTS = 32
  27. #为同一网站的请求配置延迟（默认值：0）
  28. # See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
  29. # See also autothrottle settings and docs
  30. #DOWNLOAD_DELAY = 3 下载器在下载同一个网站下一个页面前需要等待的时间,该选项可以用来限制爬取速度,减轻服务器压力。同时也支持小数:0.25 以秒为单位
  33. #下载延迟设置只有一个有效
  34. #CONCURRENT_REQUESTS_PER_DOMAIN = 16 对单个网站进行并发请求的最大值。
  35. #CONCURRENT_REQUESTS_PER_IP = 16 对单个IP进行并发请求的最大值。如果非0,则忽略 CONCURRENT_REQUESTS_PER_DOMAIN 设定,使用该设定。 也就是说,并发限制将针对IP,而不是网站。该设定也影响 DOWNLOAD_DELAY: 如果 CONCURRENT_REQUESTS_PER_IP 非0,下载延迟应用在IP而不是网站上。
  37. #禁用Cookie（默认情况下启用）
  38. #COOKIES_ENABLED = False
  40. #禁用Telnet控制台（默认启用）
  41. #TELNETCONSOLE_ENABLED = False
  43. #覆盖默认请求标头：
  44. #DEFAULT_REQUEST_HEADERS = {
  45. # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  46. # 'Accept-Language': 'en',
  47. #}
  49. #启用或禁用蜘蛛中间件
  50. # See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
  51. #SPIDER_MIDDLEWARES = {
  52. # 'demo1.middlewares.Demo1SpiderMiddleware': 543,
  53. #}
  55. #启用或禁用下载器中间件
  56. # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
  57. #DOWNLOADER_MIDDLEWARES = {
  58. # 'demo1.middlewares.MyCustomDownloaderMiddleware': 543,
  59. #}
  61. #启用或禁用扩展程序
  62. # See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
  63. #EXTENSIONS = {
  64. # 'scrapy.extensions.telnet.TelnetConsole': None,
  65. #}
  67. #配置项目管道
  68. # See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
  69. #ITEM_PIPELINES = {
  70. # 'demo1.pipelines.Demo1Pipeline': 300,
  71. #}
  73. #启用和配置AutoThrottle扩展（默认情况下禁用）
  74. # See http://doc.scrapy.org/en/latest/topics/autothrottle.html
  75. #AUTOTHROTTLE_ENABLED = True
  77. #初始下载延迟
  78. #AUTOTHROTTLE_START_DELAY = 5
  80. #在高延迟的情况下设置的最大下载延迟
  81. #AUTOTHROTTLE_MAX_DELAY = 60
  84. #Scrapy请求的平均数量应该并行发送每个远程服务器
  85. #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
  87. #启用显示所收到的每个响应的调节统计信息：
  88. #AUTOTHROTTLE_DEBUG = False
  90. #启用和配置HTTP缓存（默认情况下禁用）
  91. # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
  92. #HTTPCACHE_ENABLED = True
  93. #HTTPCACHE_EXPIRATION_SECS = 0
  94. #HTTPCACHE_DIR = 'httpcache'
  95. #HTTPCACHE_IGNORE_HTTP_CODES = []
  96. #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

解释几个参数：

ROBOTSTXT_OBEY = True-----------是否遵守robots.txt

CONCURRENT_REQUESTS = 16-----------开启线程数量，默认16

AUTOTHROTTLE_START_DELAY = 3-----------开始下载时限速并延迟时间

AUTOTHROTTLE_MAX_DELAY = 60-----------高并发请求时最大延迟时间

最底下的几个：是否启用在本地缓存，如果开启会优先读取本地缓存，从而加快爬取速度，视情况而定

HTTPCACHE_ENABLED = True

HTTPCACHE_EXPIRATION_SECS = 0

HTTPCACHE_DIR = 'httpcache'

HTTPCACHE_IGNORE_HTTP_CODES = []

HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

以上几个可以视项目需要开启，但是有两个参数最好每次都开启，而每次都是项目文件手动开启不免有些麻烦，最好是项目创建后就自动开启


#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

这个是浏览器请求头，很多网站都会检查客户端的headers，比如豆瓣就是每一个请求都检查headers的user_agent，否则只会返回403，可以开启

#USER_AGENT = 'Chirco (+http://www.yourdomain.com)'

这个是至关重要的，大部分服务器在请求快了会首先检查User_Agent,而scrapy默认的浏览器头是scrapy1.1 我们需要开启并且修改成浏览器头，如：Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1

但是最好是这个USER-AGENT会随机自动更换最好了。

下面的代码可以从预先定义的user-agent的列表中随机选择一个来采集不同的页面 

在settings.py中添加以下代码

  1. DOWNLOADER_MIDDLEWARES = {
  2. 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware' : None,
  3. 'randoms.rotate_useragent.RotateUserAgentMiddleware' :400
  4. }

rotate_useragent的代码为:

  1. # -*- coding: utf-8 -*-
  2. import random
  3. from scrapy.contrib.downloadermiddleware.useragent import UserAgentMiddleware
  5. class RotateUserAgentMiddleware(UserAgentMiddleware):
  6. def __init__(self, user_agent=''):
  7. self.user_agent = user_agent
  9. def process_request(self, request, spider):
  10. #这句话用于随机选择user-agent
  11. ua = random.choice(self.user_agent_list)
  12. if ua:
  13. print('User-Agent:'+ua)
  14. request.headers.setdefault('User-Agent', ua)
  16. #the default user_agent_list composes chrome,I E,firefox,Mozilla,opera,netscape
  17. user_agent_list = [\
  18. "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1"\
  19. "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",\
  20. "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",\
  21. "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",\
  22. "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",\
  23. "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",\
  24. "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",\
  25. "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",\
  26. "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",\
  27. "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",\
  28. "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",\
  29. "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",\
  30. "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",\
  31. "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",\
  32. "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",\
  33. "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",\
  34. "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",\
  35. "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
  36. ]


运行爬虫可以看到信息:

  1. 2017-04-16 00:07:40 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
  2. User-Agent:Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3
  3. 2017-04-16 00:07:40 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.jianshu.com/robots.txt> from <GET http://jianshu.com/robots.txt>
  4. User-Agent:Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1
  5. 2017-04-16 00:07:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.jianshu.com/robots.txt> (referer: None)
  6. User-Agent:Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24
  7. 2017-04-16 00:07:41 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.jianshu.com/> from <GET http://jianshu.com/>
  8. User-Agent:Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3

也可以把 user_agent_list放到 settings文件中去:


在rotate_useragent文件中加入一行代码

from randoms.settings import user_agent_list

运行效果如下:


完整的示例:http://download.csdn.net/detail/u011781521/9815390

猜你喜欢

转载自blog.csdn.net/rookie_is_me/article/details/80939708
今日推荐
国产云输入法——仅华为无云端数据上传安全问题
开源日报 | 工业开源项目OGG 1.0；姐姐，你要和我一起配置火狐吗；苹果AI遥遥落后？Fedora 40
开放签电子签章：停止新增，优化体验，前进更进（五一假期前工作）
开源日报 | 中学生开源前端动画引擎；全球首个Llama3 8B中文版开源模型；联想电脑恐出局；Linus讽刺AI炒作
“百模大战”必有一战 | 2024中国“百模大战”竞争格局分析
最强开源大模型 Llama 3 上架 Gitee AI
周排行
自媒体文章如何提高原创度以及如何检测原创度
开启qq邮箱的smtp服务
Qt程序单次启动（QSingleApplication类）
国外的外包网站
更新IDEA主题——放飞代码风格
cocos2dx 实现搓牌效果（翻牌效果），包括铺平动画
dict和json之间的互相转换
angular的一些思考
. Fibonacci数列是这样定义的： F[0] = 0 F[1] = 1 for each i ≥ 2: F[i] = F[i-1] + F[i-2] 因此，Fibonacci数列就形如：0, 1
洛谷P1064 金明的预算方案
每日归档
更多
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)
2024-04-18(0)
2024-04-17(5)
2024-04-16(70)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022