【scrapy】爬虫中报Forbidden by robots.txt

需要在setting.py里找到ROBOTSTXT_OBEY并设为false

来源：https://blog.csdn.net/yimingsilence/article/details/52119720