Pyppeteer和Flask问题,服务器部署Requests_html问题,多线程调用pyppeteer或requests_html问题

Pyppeteer和Flask问题,服务器部署Requests_html问题,多线程调用pyppeteer或requests_html问题

flask的debug模式下调用pyppeteer的冲突
多线程调用pyppeteer或requests_html问题
centos 8部署pyppeteer和requests_html的问题

1. flask 运行 Pyppeteer 报错 “signal only works in main thread”

puppeteer在初始化launch时，必须增加以下的参数【handleSIGINT=False, handleSIGTERM=False, handleSIGHUP=False】才能使得flask的run-debug模式启动也能正常运行！

  start_parm = {
    
    
        "handleSIGINT": False,
        "handleSIGTERM": False,
        "handleSIGHUP": False,

        # 关闭无头浏览器
        "headless": True,
        "args": [
            '--no-sandbox',  # 关闭沙盒模式
            # '--proxy-server='proxies  # 代理
        ],
    }

    # 创建浏览器对象，可以传入 字典形式参数
    browser = await launch(**start_parm)

参考链接:
	https://blog.csdn.net/weixin_43343144/article/details/110203479

2. 多线程调用pyppeteer或requests_html问题

pyppeteer:

loop1 = asyncio.new_event_loop()
asyncio.set_event_loop(loop1)
return asyncio.get_event_loop().run_until_complete(main(url, headers, proxies))

requests_html:

loop1 = asyncio.new_event_loop()
asyncio.set_event_loop(loop1)
session = HTMLSession2()
r = session.get(url)
r.html.render(sleep=.1)
print(r.html.html)

3. centos 8部署 Pyppeteer和requeests_html 报错 “pyppeteer.errors.BrowserError: Browser closed unexpectedly”

先安装 requests_html 和 pyppeteer 模块

1. pip3 install requests_html==0.10.0
2. pip3 install pyppeteer==0.2.5

运行下方代码安装puppeteer

from requests_html import HTMLSession
session = HTMLSession()
res = session.get("http://sec.didichuxing.com/present")
res.html.render() # 动态渲染页面
print(res.html.html) # 输出渲染之后的页面

安装依赖配置

3. yum install nss
4. sudo yum install pango.x86_64 libXcomposite.x86_64 libXcursor.x86_64 libXdamage.x86_64 libXext.x86_64 libXi.x86_64 libXtst.x86_64 cups-libs.x86_64 libXScrnSaver.x86_64 libXrandr.x86_64 GConf2.x86_64 alsa-lib.x86_64 atk.x86_64 gtk3.x86_64 xorg-x11-fonts-100dpi xorg-x11-fonts-75dpi xorg-x11-utils xorg-x11-fonts-cyrillic xorg-x11-fonts-Type1 xorg-x11-fonts-misc
# 如果是centos 7系统使用
5. yum update nss -y
# 如果是centos 8系统使用
5. yum update nss chrony

参考链接:
	https://blog.csdn.net/qq_26870933/article/details/101288399?utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7EBlogCommendFromMachineLearnPai2%7Edefault-4.control&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7EBlogCommendFromMachineLearnPai2%7Edefault-4.control
	https://blog.csdn.net/wgPython/article/details/102519342